[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:14:52 dist-test-slave-hndd krb5kdc[2498](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:14:52 dist-test-slave-hndd krb5kdc[2498](info): set up 2 sockets
Sep 26 06:14:52 dist-test-slave-hndd krb5kdc[2498](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 06:14:53.433808 2491 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Sep 26 06:14:53 dist-test-slave-hndd krb5kdc[2498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:14:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:14:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
I20260926 06:14:53.569464 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:44837
--webserver_interface=127.2.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:34491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.110.254:44837
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:14:53.654965 2511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:14:53.655195 2511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:14:53.655220 2511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:14:53.656647 2511 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:14:53.656694 2511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:14:53.656708 2511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:14:53.656721 2511 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:14:53.656761 2511 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:14:53.658406 2511 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:34491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.110.254:44837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:44837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:14:53.658687 2511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:14:53.658949 2511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:14:53.664271 2517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:14:53.664378 2511 server_base.cc:1034] running on GCE node
W20260926 06:14:53.664260 2516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:14:53.664243 2519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:14:53.664829 2511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:14:53.665143 2511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:14:53.666311 2511 hybrid_clock.cc:648] HybridClock initialized: now 1790403293666300 us; error 38 us; skew 500 ppm
Sep 26 06:14:53 dist-test-slave-hndd krb5kdc[2498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403293, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:14:53.674218 2511 init.cc:377] Logged in from keytab as kudu/127.2.110.254@KRBTEST.COM (short username kudu)
I20260926 06:14:53.674690 2511 webserver.cc:460] Webserver started at http://127.2.110.254:34719/ using document root <none> and password file <none>
I20260926 06:14:53.674961 2511 fs_manager.cc:359] Metadata directory not provided
I20260926 06:14:53.675034 2511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:14:53.675195 2511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:14:53.676321 2511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/master-0/data/instance:
uuid: "0faa524a8d4b442496c32fec9aa4096d"
format_stamp: "Formatted at 2026-09-26 06:14:53 on dist-test-slave-hndd"
server_key: "00b0885400bafd2e12d5763ccaf00da2"
server_key_iv: "423f4147fee1a78d9cd436dc9f19a9e3"
server_key_version: "encryptionkey@0"
I20260926 06:14:53.676699 2511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/master-0/wal/instance:
uuid: "0faa524a8d4b442496c32fec9aa4096d"
format_stamp: "Formatted at 2026-09-26 06:14:53 on dist-test-slave-hndd"
server_key: "00b0885400bafd2e12d5763ccaf00da2"
server_key_iv: "423f4147fee1a78d9cd436dc9f19a9e3"
server_key_version: "encryptionkey@0"
I20260926 06:14:53.678243 2511 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260926 06:14:53.679090 2526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:14:53.679291 2511 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:14:53.679399 2511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "0faa524a8d4b442496c32fec9aa4096d"
format_stamp: "Formatted at 2026-09-26 06:14:53 on dist-test-slave-hndd"
server_key: "00b0885400bafd2e12d5763ccaf00da2"
server_key_iv: "423f4147fee1a78d9cd436dc9f19a9e3"
server_key_version: "encryptionkey@0"
I20260926 06:14:53.679487 2511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:14:53.692032 2511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:14:53.694470 2511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:14:53.694648 2511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:14:53.698552 2511 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.254:44837
I20260926 06:14:53.698588 2578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.254:44837 every 8 connection(s)
I20260926 06:14:53.698941 2511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:14:53.699877 2579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:14:53.702324 2579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0faa524a8d4b442496c32fec9aa4096d: Bootstrap starting.
I20260926 06:14:53.702883 2579 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0faa524a8d4b442496c32fec9aa4096d: Neither blocks nor log segments found. Creating new log.
I20260926 06:14:53.703140 2579 log.cc:824] T 00000000000000000000000000000000 P 0faa524a8d4b442496c32fec9aa4096d: Log is configured to *not* fsync() on all Append() calls
I20260926 06:14:53.703811 2579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0faa524a8d4b442496c32fec9aa4096d: No bootstrap required, opened a new log
I20260926 06:14:53.704780 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 2511
I20260926 06:14:53.704911 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/master-0/wal/instance
I20260926 06:14:53.705732 2491 external_mini_cluster.cc:1428] Setting key 2a9aa27e2a90d70438ff5c16e0da2788
I20260926 06:14:53.706127 2579 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0faa524a8d4b442496c32fec9aa4096d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0faa524a8d4b442496c32fec9aa4096d" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 44837 } }
I20260926 06:14:53.706291 2579 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0faa524a8d4b442496c32fec9aa4096d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:14:53.706341 2579 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0faa524a8d4b442496c32fec9aa4096d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0faa524a8d4b442496c32fec9aa4096d, State: Initialized, Role: FOLLOWER
I20260926 06:14:53.706497 2579 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0faa524a8d4b442496c32fec9aa4096d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0faa524a8d4b442496c32fec9aa4096d" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 44837 } }
I20260926 06:14:53.706619 2579 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0faa524a8d4b442496c32fec9aa4096d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:14:53.706689 2579 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0faa524a8d4b442496c32fec9aa4096d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:14:53.706753 2579 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0faa524a8d4b442496c32fec9aa4096d [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:14:53.707734 2579 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0faa524a8d4b442496c32fec9aa4096d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0faa524a8d4b442496c32fec9aa4096d" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 44837 } }
I20260926 06:14:53.707827 2579 leader_election.cc:302] T 00000000000000000000000000000000 P 0faa524a8d4b442496c32fec9aa4096d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0faa524a8d4b442496c32fec9aa4096d; no voters:
I20260926 06:14:53.708024 2579 leader_election.cc:288] T 00000000000000000000000000000000 P 0faa524a8d4b442496c32fec9aa4096d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:14:53.708065 2584 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0faa524a8d4b442496c32fec9aa4096d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:14:53.708236 2584 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0faa524a8d4b442496c32fec9aa4096d [term 1 LEADER]: Becoming Leader. State: Replica: 0faa524a8d4b442496c32fec9aa4096d, State: Running, Role: LEADER
I20260926 06:14:53.708312 2579 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0faa524a8d4b442496c32fec9aa4096d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:14:53.708396 2584 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0faa524a8d4b442496c32fec9aa4096d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0faa524a8d4b442496c32fec9aa4096d" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 44837 } }
I20260926 06:14:53.708808 2584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0faa524a8d4b442496c32fec9aa4096d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0faa524a8d4b442496c32fec9aa4096d. Latest consensus state: current_term: 1 leader_uuid: "0faa524a8d4b442496c32fec9aa4096d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0faa524a8d4b442496c32fec9aa4096d" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 44837 } } }
I20260926 06:14:53.709024 2584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0faa524a8d4b442496c32fec9aa4096d [sys.catalog]: This master's current role is: LEADER
I20260926 06:14:53.708966 2586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0faa524a8d4b442496c32fec9aa4096d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0faa524a8d4b442496c32fec9aa4096d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0faa524a8d4b442496c32fec9aa4096d" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 44837 } } }
I20260926 06:14:53.709271 2586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0faa524a8d4b442496c32fec9aa4096d [sys.catalog]: This master's current role is: LEADER
W20260926 06:14:53.710143 2600 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0faa524a8d4b442496c32fec9aa4096d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 06:14:53.710216 2600 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 06:14:53.710323 2593 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:14:53.710485 2593 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:14:53.711864 2593 catalog_manager.cc:1347] Generated new cluster ID: 22eec7b76a13440cac76a99b0a455c10
I20260926 06:14:53.711923 2593 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 06:14:53 dist-test-slave-hndd krb5kdc[2498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:14:53.724968 2582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:14:53.706631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35654 (local address 127.2.110.254:44837)
0926 06:14:53.706830 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 06:14:53.706840 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 06:14:53.707149 (+ 309us) server_negotiation.cc:407] Connection header received
0926 06:14:53.709373 (+ 2224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:14:53.709381 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:14:53.709442 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:14:53.709559 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:14:53.711975 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:14:53.713804 (+ 1829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:14:53.715341 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:14:53.715964 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:14:53.716065 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:14:53.720616 (+ 4551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:14:53.720621 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:14:53.720622 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:14:53.720637 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:14:53.722760 (+ 2123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:14:53.723312 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:14:53.723315 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:14:53.723316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:14:53.723436 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:14:53.723738 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:14:53.723739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:14:53.723739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:14:53.724082 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:14:53.724223 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 06:14:53.724540 (+ 317us) server_negotiation.cc:299] Negotiation successful
0926 06:14:53.724689 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":97,"threads_started":1}
I20260926 06:14:53.740667 2593 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:14:53.741266 2593 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:14:53.747957 2593 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0faa524a8d4b442496c32fec9aa4096d: Generated new TSK 0
I20260926 06:14:53.748221 2593 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab.
Entry for principal kudu/127.2.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab.
I20260926 06:14:53.784013 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.193:0
--local_ip_for_outbound_sockets=127.2.110.193
--webserver_interface=127.2.110.193
--webserver_port=0
--tserver_master_addrs=127.2.110.254:44837
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab
--principal=kudu/127.2.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:34491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:14:53.869918 2605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:14:53.870143 2605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:14:53.870170 2605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:14:53.871435 2605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:14:53.871484 2605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:14:53.871515 2605 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.193
I20260926 06:14:53.873066 2605 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:34491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab
--principal=kudu/127.2.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.110.193
--webserver_port=0
--tserver_master_addrs=127.2.110.254:44837
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.193
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:14:53.873381 2605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:14:53.873656 2605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:14:53.879201 2610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:14:53.879318 2611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:14:53.879336 2605 server_base.cc:1034] running on GCE node
W20260926 06:14:53.879240 2613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:14:53.879673 2605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:14:53.879959 2605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:14:53.881147 2605 hybrid_clock.cc:648] HybridClock initialized: now 1790403293881143 us; error 43 us; skew 500 ppm
Sep 26 06:14:53 dist-test-slave-hndd krb5kdc[2498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403293, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:14:53.884191 2605 init.cc:377] Logged in from keytab as kudu/127.2.110.193@KRBTEST.COM (short username kudu)
I20260926 06:14:53.884717 2605 webserver.cc:460] Webserver started at http://127.2.110.193:36213/ using document root <none> and password file <none>
I20260926 06:14:53.884979 2605 fs_manager.cc:359] Metadata directory not provided
I20260926 06:14:53.885056 2605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:14:53.885231 2605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:14:53.886581 2605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/ts-0/data/instance:
uuid: "c476cda4940740cfb2b298d86226c617"
format_stamp: "Formatted at 2026-09-26 06:14:53 on dist-test-slave-hndd"
server_key: "3daaf1aff1d976651b39d82c87c5ef64"
server_key_iv: "1f31abbf274d8e9a9dd99768bf412422"
server_key_version: "encryptionkey@0"
I20260926 06:14:53.887028 2605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/ts-0/wal/instance:
uuid: "c476cda4940740cfb2b298d86226c617"
format_stamp: "Formatted at 2026-09-26 06:14:53 on dist-test-slave-hndd"
server_key: "3daaf1aff1d976651b39d82c87c5ef64"
server_key_iv: "1f31abbf274d8e9a9dd99768bf412422"
server_key_version: "encryptionkey@0"
I20260926 06:14:53.888585 2605 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:14:53.889606 2620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:14:53.889816 2605 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:14:53.889938 2605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/ts-0/wal
uuid: "c476cda4940740cfb2b298d86226c617"
format_stamp: "Formatted at 2026-09-26 06:14:53 on dist-test-slave-hndd"
server_key: "3daaf1aff1d976651b39d82c87c5ef64"
server_key_iv: "1f31abbf274d8e9a9dd99768bf412422"
server_key_version: "encryptionkey@0"
I20260926 06:14:53.890026 2605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:14:53.909996 2605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:14:53.912722 2605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:14:53.912948 2605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:14:53.913244 2605 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:14:53.913697 2605 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:14:53.913759 2605 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:14:53.913806 2605 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:14:53.913856 2605 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:14:53.920892 2605 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.193:33803
I20260926 06:14:53.920979 2733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.193:33803 every 8 connection(s)
I20260926 06:14:53.921371 2605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
Sep 26 06:14:53 dist-test-slave-hndd krb5kdc[2498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403293, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.193@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:14:53.930215 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 2605
I20260926 06:14:53.930300 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/ts-0/wal/instance
I20260926 06:14:53.930510 2491 external_mini_cluster.cc:1428] Setting key 1780db85dbf35c4f3113f206adefc54e
I20260926 06:14:53.931773 2736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:14:53.922164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:44837 (local address 127.2.110.193:43661)
0926 06:14:53.922326 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 06:14:53.922371 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 06:14:53.922912 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:14:53.923149 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:14:53.923153 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:14:53.923748 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:14:53.924010 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:14:53.924017 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:14:53.924933 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:14:53.924934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:14:53.925851 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:14:53.925854 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:14:53.926269 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:14:53.926270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:14:53.926375 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:14:53.926904 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:14:53.926912 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:14:53.928449 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:14:53.930136 (+ 1687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:14:53.930143 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:14:53.930145 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:14:53.930432 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:14:53.930673 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:14:53.930676 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:14:53.930676 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:14:53.930848 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:14:53.931188 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:14:53.931196 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:14:53.931375 (+ 179us) client_negotiation.cc:769] Sending connection context
0926 06:14:53.931467 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 06:14:53.931606 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":86,"threads_started":1}
I20260926 06:14:53.931783 2582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:14:53.922218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:43661 (local address 127.2.110.254:44837)
0926 06:14:53.922312 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0926 06:14:53.922314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:14:53.922481 (+ 167us) server_negotiation.cc:407] Connection header received
0926 06:14:53.923011 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:14:53.923012 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:14:53.923027 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:14:53.923096 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:14:53.924159 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:14:53.924828 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:14:53.925959 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:14:53.926175 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:14:53.926256 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:14:53.928568 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:14:53.928570 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:14:53.928571 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:14:53.928580 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:14:53.930047 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:14:53.930510 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:14:53.930511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:14:53.930511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:14:53.930583 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:14:53.930936 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:14:53.930937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:14:53.930937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:14:53.931067 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:14:53.931164 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 06:14:53.931524 (+ 360us) server_negotiation.cc:299] Negotiation successful
0926 06:14:53.931625 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 06:14:53.932469 2734 heartbeater.cc:347] Connected to a master server at 127.2.110.254:44837
I20260926 06:14:53.932582 2734 heartbeater.cc:464] Registering TS with master...
I20260926 06:14:53.932816 2734 heartbeater.cc:511] Master 127.2.110.254:44837 requested a full tablet report, sending...
I20260926 06:14:53.933836 2543 ts_manager.cc:194] Registered new tserver with Master: c476cda4940740cfb2b298d86226c617 (127.2.110.193:33803)
I20260926 06:14:53.934604 2543 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.193@KRBTEST.COM'} at 127.2.110.193:43661
WARNING: no policy specified for kudu/127.2.110.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab.
Entry for principal kudu/127.2.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab.
I20260926 06:14:53.961637 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.194:0
--local_ip_for_outbound_sockets=127.2.110.194
--webserver_interface=127.2.110.194
--webserver_port=0
--tserver_master_addrs=127.2.110.254:44837
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab
--principal=kudu/127.2.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:34491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:14:54.048322 2739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:14:54.048604 2739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:14:54.048633 2739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:14:54.049958 2739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:14:54.050009 2739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:14:54.050040 2739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.194
I20260926 06:14:54.051597 2739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:34491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab
--principal=kudu/127.2.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.110.194
--webserver_port=0
--tserver_master_addrs=127.2.110.254:44837
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.194
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:14:54.051856 2739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:14:54.052115 2739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:14:54.058000 2744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:14:54.058007 2745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:14:54.058012 2747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:14:54.058228 2739 server_base.cc:1034] running on GCE node
I20260926 06:14:54.058511 2739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:14:54.058775 2739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:14:54.059965 2739 hybrid_clock.cc:648] HybridClock initialized: now 1790403294059947 us; error 33 us; skew 500 ppm
Sep 26 06:14:54 dist-test-slave-hndd krb5kdc[2498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403294, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:14:54.062954 2739 init.cc:377] Logged in from keytab as kudu/127.2.110.194@KRBTEST.COM (short username kudu)
I20260926 06:14:54.063503 2739 webserver.cc:460] Webserver started at http://127.2.110.194:41309/ using document root <none> and password file <none>
I20260926 06:14:54.063750 2739 fs_manager.cc:359] Metadata directory not provided
I20260926 06:14:54.063822 2739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:14:54.063984 2739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:14:54.065217 2739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/ts-1/data/instance:
uuid: "df3ba5d390534d95acd2c0f00c38860d"
format_stamp: "Formatted at 2026-09-26 06:14:54 on dist-test-slave-hndd"
server_key: "b3934c9b8666dc477404d30af612ebbe"
server_key_iv: "abc9c05fb53f593438c45336d9a97b09"
server_key_version: "encryptionkey@0"
I20260926 06:14:54.065714 2739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/ts-1/wal/instance:
uuid: "df3ba5d390534d95acd2c0f00c38860d"
format_stamp: "Formatted at 2026-09-26 06:14:54 on dist-test-slave-hndd"
server_key: "b3934c9b8666dc477404d30af612ebbe"
server_key_iv: "abc9c05fb53f593438c45336d9a97b09"
server_key_version: "encryptionkey@0"
I20260926 06:14:54.067106 2739 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:14:54.067999 2754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:14:54.068382 2739 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:14:54.068539 2739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/ts-1/wal
uuid: "df3ba5d390534d95acd2c0f00c38860d"
format_stamp: "Formatted at 2026-09-26 06:14:54 on dist-test-slave-hndd"
server_key: "b3934c9b8666dc477404d30af612ebbe"
server_key_iv: "abc9c05fb53f593438c45336d9a97b09"
server_key_version: "encryptionkey@0"
I20260926 06:14:54.068631 2739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:14:54.093837 2739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:14:54.096273 2739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:14:54.096464 2739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:14:54.096750 2739 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:14:54.097096 2739 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:14:54.097155 2739 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:14:54.097200 2739 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:14:54.097250 2739 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:14:54.104686 2739 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.194:43227
I20260926 06:14:54.104739 2867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.194:43227 every 8 connection(s)
I20260926 06:14:54.105113 2739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
I20260926 06:14:54.107499 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 2739
I20260926 06:14:54.107626 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/ts-1/wal/instance
I20260926 06:14:54.107875 2491 external_mini_cluster.cc:1428] Setting key 99b966b1ac4cf66d5e2ef920dc38c194
Sep 26 06:14:54 dist-test-slave-hndd krb5kdc[2498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403294, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.194@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:14:54.116846 2582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:14:54.106229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:44673 (local address 127.2.110.254:44837)
0926 06:14:54.106377 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 06:14:54.106380 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:14:54.106401 (+ 21us) server_negotiation.cc:407] Connection header received
0926 06:14:54.107111 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:14:54.107112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:14:54.107127 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:14:54.107323 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:14:54.109004 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:14:54.109711 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:14:54.110850 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:14:54.111070 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:14:54.111172 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:14:54.113819 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:14:54.113821 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:14:54.113821 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:14:54.113832 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:14:54.115346 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:14:54.115825 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:14:54.115826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:14:54.115827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:14:54.115888 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:14:54.116159 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:14:54.116160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:14:54.116161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:14:54.116284 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:14:54.116362 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 06:14:54.116645 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 06:14:54.116741 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260926 06:14:54.117072 2870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:14:54.105907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:44837 (local address 127.2.110.194:44673)
0926 06:14:54.106084 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 06:14:54.106129 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 06:14:54.106935 (+ 806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:14:54.107430 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:14:54.107436 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:14:54.108509 (+ 1073us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:14:54.108876 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:14:54.108884 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:14:54.109812 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:14:54.109814 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:14:54.110733 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:14:54.110736 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:14:54.111182 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:14:54.111183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:14:54.111347 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:14:54.111951 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:14:54.111962 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:14:54.113705 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:14:54.115479 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:14:54.115486 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:14:54.115489 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:14:54.115742 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:14:54.115961 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:14:54.115963 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:14:54.115963 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:14:54.116082 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:14:54.116353 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:14:54.116360 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:14:54.116509 (+ 149us) client_negotiation.cc:769] Sending connection context
0926 06:14:54.116627 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 06:14:54.116820 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":74,"threads_started":1}
WARNING: no policy specified for kudu/127.2.110.195@KRBTEST.COM; defaulting to no policy
I20260926 06:14:54.117533 2868 heartbeater.cc:347] Connected to a master server at 127.2.110.254:44837
I20260926 06:14:54.117646 2868 heartbeater.cc:464] Registering TS with master...
I20260926 06:14:54.117828 2868 heartbeater.cc:511] Master 127.2.110.254:44837 requested a full tablet report, sending...
I20260926 06:14:54.118287 2543 ts_manager.cc:194] Registered new tserver with Master: df3ba5d390534d95acd2c0f00c38860d (127.2.110.194:43227)
I20260926 06:14:54.118644 2543 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.194@KRBTEST.COM'} at 127.2.110.194:44673
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab.
Entry for principal kudu/127.2.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab.
I20260926 06:14:54.136142 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.195:0
--local_ip_for_outbound_sockets=127.2.110.195
--webserver_interface=127.2.110.195
--webserver_port=0
--tserver_master_addrs=127.2.110.254:44837
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab
--principal=kudu/127.2.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:34491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:14:54.225895 2873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:14:54.226119 2873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:14:54.226147 2873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:14:54.227439 2873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:14:54.227486 2873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:14:54.227516 2873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.195
I20260926 06:14:54.229044 2873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:34491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab
--principal=kudu/127.2.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.110.195
--webserver_port=0
--tserver_master_addrs=127.2.110.254:44837
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.195
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:14:54.229306 2873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:14:54.229635 2873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 06:14:54.235294 2873 server_base.cc:1034] running on GCE node
W20260926 06:14:54.235275 2878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:14:54.235277 2881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:14:54.235270 2879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:14:54.235736 2873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:14:54.235956 2873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:14:54.237110 2873 hybrid_clock.cc:648] HybridClock initialized: now 1790403294237087 us; error 39 us; skew 500 ppm
Sep 26 06:14:54 dist-test-slave-hndd krb5kdc[2498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403294, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:14:54.239817 2873 init.cc:377] Logged in from keytab as kudu/127.2.110.195@KRBTEST.COM (short username kudu)
I20260926 06:14:54.240336 2873 webserver.cc:460] Webserver started at http://127.2.110.195:34995/ using document root <none> and password file <none>
I20260926 06:14:54.240600 2873 fs_manager.cc:359] Metadata directory not provided
I20260926 06:14:54.240669 2873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:14:54.240818 2873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:14:54.241973 2873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/ts-2/data/instance:
uuid: "c2387baf9c084deaacbc1465d01060f5"
format_stamp: "Formatted at 2026-09-26 06:14:54 on dist-test-slave-hndd"
server_key: "e25738fb7e3b2829bf2d10da3e7d964a"
server_key_iv: "1f0de7170dc0beb8c5c6c9a8125b482b"
server_key_version: "encryptionkey@0"
I20260926 06:14:54.242342 2873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/ts-2/wal/instance:
uuid: "c2387baf9c084deaacbc1465d01060f5"
format_stamp: "Formatted at 2026-09-26 06:14:54 on dist-test-slave-hndd"
server_key: "e25738fb7e3b2829bf2d10da3e7d964a"
server_key_iv: "1f0de7170dc0beb8c5c6c9a8125b482b"
server_key_version: "encryptionkey@0"
I20260926 06:14:54.243760 2873 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 06:14:54.244784 2888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:14:54.245204 2873 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:14:54.245389 2873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/ts-2/wal
uuid: "c2387baf9c084deaacbc1465d01060f5"
format_stamp: "Formatted at 2026-09-26 06:14:54 on dist-test-slave-hndd"
server_key: "e25738fb7e3b2829bf2d10da3e7d964a"
server_key_iv: "1f0de7170dc0beb8c5c6c9a8125b482b"
server_key_version: "encryptionkey@0"
I20260926 06:14:54.245488 2873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:14:54.258142 2873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:14:54.261799 2873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:14:54.261937 2873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:14:54.262193 2873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:14:54.262539 2873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:14:54.262573 2873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:14:54.262598 2873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:14:54.262639 2873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:14:54.271679 2873 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.195:35357
I20260926 06:14:54.271777 3001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.195:35357 every 8 connection(s)
I20260926 06:14:54.272281 2873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
Sep 26 06:14:54 dist-test-slave-hndd krb5kdc[2498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403294, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.195@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:14:54.281877 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 2873
I20260926 06:14:54.281991 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403292369990-2491-0/minicluster-data/ts-2/wal/instance
I20260926 06:14:54.282188 2491 external_mini_cluster.cc:1428] Setting key c87d12d15411020395073af01457bc60
I20260926 06:14:54.283535 2582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:14:54.273298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:49877 (local address 127.2.110.254:44837)
0926 06:14:54.273452 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 06:14:54.273454 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:14:54.273650 (+ 196us) server_negotiation.cc:407] Connection header received
0926 06:14:54.274236 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:14:54.274238 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:14:54.274251 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:14:54.274297 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:14:54.275339 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:14:54.275901 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:14:54.276986 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:14:54.277198 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:14:54.277297 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:14:54.279688 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:14:54.279690 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:14:54.279690 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:14:54.279699 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:14:54.281525 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:14:54.281967 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:14:54.281969 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:14:54.281969 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:14:54.282039 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:14:54.282460 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:14:54.282462 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:14:54.282462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:14:54.282749 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:14:54.282868 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 06:14:54.283295 (+ 427us) server_negotiation.cc:299] Negotiation successful
0926 06:14:54.283397 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260926 06:14:54.283720 3004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:14:54.273245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:44837 (local address 127.2.110.195:49877)
0926 06:14:54.273495 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 06:14:54.273558 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 06:14:54.274106 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:14:54.274384 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:14:54.274388 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:14:54.275019 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:14:54.275224 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:14:54.275229 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:14:54.276008 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:14:54.276009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:14:54.276871 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:14:54.276873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:14:54.277305 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:14:54.277306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:14:54.277481 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:14:54.278056 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:14:54.278065 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:14:54.279564 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:14:54.281624 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:14:54.281631 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:14:54.281634 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:14:54.281878 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:14:54.282116 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:14:54.282117 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:14:54.282118 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:14:54.282302 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:14:54.282861 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:14:54.282868 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:14:54.283059 (+ 191us) client_negotiation.cc:769] Sending connection context
0926 06:14:54.283219 (+ 160us) client_negotiation.cc:240] Negotiation successful
0926 06:14:54.283434 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":131,"threads_started":1}
I20260926 06:14:54.284279 3002 heartbeater.cc:347] Connected to a master server at 127.2.110.254:44837
I20260926 06:14:54.284493 3002 heartbeater.cc:464] Registering TS with master...
I20260926 06:14:54.284838 3002 heartbeater.cc:511] Master 127.2.110.254:44837 requested a full tablet report, sending...
I20260926 06:14:54.285555 2543 ts_manager.cc:194] Registered new tserver with Master: c2387baf9c084deaacbc1465d01060f5 (127.2.110.195:35357)
I20260926 06:14:54.286036 2543 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.195@KRBTEST.COM'} at 127.2.110.195:49877
I20260926 06:14:54.295572 2491 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 06:14:54 dist-test-slave-hndd krb5kdc[2498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403294, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 06:14:54 dist-test-slave-hndd krb5kdc[2498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403294, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.110.193@KRBTEST.COM
I20260926 06:14:54.321919 3009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:14:54.312788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50450 (local address 127.2.110.193:33803)
0926 06:14:54.313066 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0926 06:14:54.313071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:14:54.313084 (+ 13us) server_negotiation.cc:407] Connection header received
0926 06:14:54.313112 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:14:54.313114 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:14:54.313129 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:14:54.313216 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:14:54.313890 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:14:54.314656 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:14:54.315474 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:14:54.315863 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:14:54.315938 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:14:54.318839 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:14:54.318850 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:14:54.318851 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:14:54.318858 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:14:54.320601 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:14:54.321065 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:14:54.321068 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:14:54.321069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:14:54.321138 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:14:54.321410 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:14:54.321411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:14:54.321411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:14:54.321579 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:14:54.321703 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 06:14:54.321766 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 06:14:54.321839 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":129,"threads_started":1}
W20260926 06:14:54.322347 2648 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50450
Sep 26 06:14:54 dist-test-slave-hndd krb5kdc[2498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 06:14:54 dist-test-slave-hndd krb5kdc[2498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.110.193@KRBTEST.COM
I20260926 06:14:54.350675 3009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:14:54.341785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50452 (local address 127.2.110.193:33803)
0926 06:14:54.341864 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 06:14:54.341867 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:14:54.342019 (+ 152us) server_negotiation.cc:407] Connection header received
0926 06:14:54.342127 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:14:54.342128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:14:54.342148 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:14:54.342357 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:14:54.342999 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:14:54.343507 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:14:54.344277 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:14:54.344497 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:14:54.344579 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:14:54.347132 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:14:54.347135 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:14:54.347135 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:14:54.347147 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:14:54.348679 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:14:54.349171 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:14:54.349172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:14:54.349173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:14:54.349464 (+ 291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:14:54.349747 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:14:54.349748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:14:54.349749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:14:54.349984 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:14:54.350338 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0926 06:14:54.350387 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 06:14:54.350497 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 06:14:54.351804 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 2605
I20260926 06:14:54.357563 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 2739
I20260926 06:14:54.363830 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 2873
I20260926 06:14:54.368920 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 2511
2026-09-26T06:14:54Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2006 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:14:54 dist-test-slave-hndd krb5kdc[3016](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:14:54 dist-test-slave-hndd krb5kdc[3016](info): set up 2 sockets
Sep 26 06:14:54 dist-test-slave-hndd krb5kdc[3016](info): commencing operation
krb5kdc: starting...
W20260926 06:14:55.420974 2491 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Sep 26 06:14:55 dist-test-slave-hndd krb5kdc[3016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:14:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:14:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
I20260926 06:14:55.548915 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:38819
--webserver_interface=127.2.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:45547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.110.254:38819
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:14:55.632740 3029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:14:55.632999 3029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:14:55.633038 3029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:14:55.634250 3029 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:14:55.634320 3029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:14:55.634346 3029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:14:55.634372 3029 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:14:55.634433 3029 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:14:55.635905 3029 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:45547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.110.254:38819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:38819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:14:55.636180 3029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:14:55.636602 3029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:14:55.641650 3035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:14:55.641649 3037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:14:55.641685 3034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:14:55.641827 3029 server_base.cc:1034] running on GCE node
I20260926 06:14:55.642133 3029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:14:55.642391 3029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:14:55.643514 3029 hybrid_clock.cc:648] HybridClock initialized: now 1790403295643493 us; error 40 us; skew 500 ppm
Sep 26 06:14:55 dist-test-slave-hndd krb5kdc[3016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403295, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:14:55.646093 3029 init.cc:377] Logged in from keytab as kudu/127.2.110.254@KRBTEST.COM (short username kudu)
I20260926 06:14:55.646553 3029 webserver.cc:460] Webserver started at http://127.2.110.254:44099/ using document root <none> and password file <none>
I20260926 06:14:55.646792 3029 fs_manager.cc:359] Metadata directory not provided
I20260926 06:14:55.646862 3029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:14:55.647035 3029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:14:55.648381 3029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/master-0/data/instance:
uuid: "a0b89bc3234c40d3a2ca000bea7742ae"
format_stamp: "Formatted at 2026-09-26 06:14:55 on dist-test-slave-hndd"
server_key: "dc84db04294d9f40ddf699a8fd798ef9"
server_key_iv: "f695fc1d850dce9ebfecfc4236212b85"
server_key_version: "encryptionkey@0"
I20260926 06:14:55.648792 3029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/master-0/wal/instance:
uuid: "a0b89bc3234c40d3a2ca000bea7742ae"
format_stamp: "Formatted at 2026-09-26 06:14:55 on dist-test-slave-hndd"
server_key: "dc84db04294d9f40ddf699a8fd798ef9"
server_key_iv: "f695fc1d850dce9ebfecfc4236212b85"
server_key_version: "encryptionkey@0"
I20260926 06:14:55.650262 3029 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:14:55.651044 3044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:14:55.651228 3029 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 06:14:55.651336 3029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "a0b89bc3234c40d3a2ca000bea7742ae"
format_stamp: "Formatted at 2026-09-26 06:14:55 on dist-test-slave-hndd"
server_key: "dc84db04294d9f40ddf699a8fd798ef9"
server_key_iv: "f695fc1d850dce9ebfecfc4236212b85"
server_key_version: "encryptionkey@0"
I20260926 06:14:55.651427 3029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:14:55.662302 3029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:14:55.664307 3029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:14:55.664489 3029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:14:55.668681 3029 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.254:38819
I20260926 06:14:55.668735 3096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.254:38819 every 8 connection(s)
I20260926 06:14:55.669107 3029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:14:55.670099 3097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:14:55.672401 3097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0b89bc3234c40d3a2ca000bea7742ae: Bootstrap starting.
I20260926 06:14:55.672998 3097 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0b89bc3234c40d3a2ca000bea7742ae: Neither blocks nor log segments found. Creating new log.
I20260926 06:14:55.673295 3097 log.cc:824] T 00000000000000000000000000000000 P a0b89bc3234c40d3a2ca000bea7742ae: Log is configured to *not* fsync() on all Append() calls
I20260926 06:14:55.673964 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 3029
I20260926 06:14:55.674049 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/master-0/wal/instance
I20260926 06:14:55.674165 3097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0b89bc3234c40d3a2ca000bea7742ae: No bootstrap required, opened a new log
I20260926 06:14:55.674211 2491 external_mini_cluster.cc:1428] Setting key f6aef12e0367b56af7dcb382d753a4d3
I20260926 06:14:55.677167 3097 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0b89bc3234c40d3a2ca000bea7742ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0b89bc3234c40d3a2ca000bea7742ae" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 38819 } }
I20260926 06:14:55.677314 3097 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0b89bc3234c40d3a2ca000bea7742ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:14:55.677413 3097 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0b89bc3234c40d3a2ca000bea7742ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0b89bc3234c40d3a2ca000bea7742ae, State: Initialized, Role: FOLLOWER
I20260926 06:14:55.677561 3097 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0b89bc3234c40d3a2ca000bea7742ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0b89bc3234c40d3a2ca000bea7742ae" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 38819 } }
I20260926 06:14:55.677644 3097 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0b89bc3234c40d3a2ca000bea7742ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:14:55.677668 3097 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0b89bc3234c40d3a2ca000bea7742ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:14:55.677733 3097 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0b89bc3234c40d3a2ca000bea7742ae [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:14:55.678329 3097 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0b89bc3234c40d3a2ca000bea7742ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0b89bc3234c40d3a2ca000bea7742ae" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 38819 } }
I20260926 06:14:55.678426 3097 leader_election.cc:302] T 00000000000000000000000000000000 P a0b89bc3234c40d3a2ca000bea7742ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0b89bc3234c40d3a2ca000bea7742ae; no voters:
I20260926 06:14:55.678658 3097 leader_election.cc:288] T 00000000000000000000000000000000 P a0b89bc3234c40d3a2ca000bea7742ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:14:55.678681 3102 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0b89bc3234c40d3a2ca000bea7742ae [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:14:55.678833 3102 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0b89bc3234c40d3a2ca000bea7742ae [term 1 LEADER]: Becoming Leader. State: Replica: a0b89bc3234c40d3a2ca000bea7742ae, State: Running, Role: LEADER
I20260926 06:14:55.678927 3102 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0b89bc3234c40d3a2ca000bea7742ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0b89bc3234c40d3a2ca000bea7742ae" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 38819 } }
I20260926 06:14:55.678929 3097 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0b89bc3234c40d3a2ca000bea7742ae [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:14:55.679239 3102 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0b89bc3234c40d3a2ca000bea7742ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0b89bc3234c40d3a2ca000bea7742ae. Latest consensus state: current_term: 1 leader_uuid: "a0b89bc3234c40d3a2ca000bea7742ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0b89bc3234c40d3a2ca000bea7742ae" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 38819 } } }
I20260926 06:14:55.679302 3102 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0b89bc3234c40d3a2ca000bea7742ae [sys.catalog]: This master's current role is: LEADER
I20260926 06:14:55.679402 3103 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0b89bc3234c40d3a2ca000bea7742ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0b89bc3234c40d3a2ca000bea7742ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0b89bc3234c40d3a2ca000bea7742ae" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 38819 } } }
I20260926 06:14:55.679461 3103 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0b89bc3234c40d3a2ca000bea7742ae [sys.catalog]: This master's current role is: LEADER
I20260926 06:14:55.679538 3106 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:14:55.680061 3106 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 06:14:55 dist-test-slave-hndd krb5kdc[3016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:14:55.681614 3119 mvcc.cc:205] Tried to move back new op lower bound from 7333491899109453824 to 7333491899101265920. Current Snapshot: MvccSnapshot[applied={T|T < 7333491899109453824}]
I20260926 06:14:55.682013 3106 catalog_manager.cc:1347] Generated new cluster ID: c2396aa1937b43d4939bfe61dfa6729b
I20260926 06:14:55.682070 3106 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:14:55.685506 3101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:14:55.674741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60212 (local address 127.2.110.254:38819)
0926 06:14:55.674932 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 06:14:55.674940 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 06:14:55.674969 (+ 29us) server_negotiation.cc:407] Connection header received
0926 06:14:55.675469 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:14:55.675476 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:14:55.675526 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:14:55.675681 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:14:55.676292 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:14:55.677173 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:14:55.678137 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:14:55.678647 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:14:55.678770 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:14:55.682003 (+ 3233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:14:55.682007 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:14:55.682007 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:14:55.682020 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:14:55.683837 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:14:55.684217 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:14:55.684220 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:14:55.684221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:14:55.684322 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:14:55.684543 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:14:55.684544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:14:55.684545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:14:55.684791 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:14:55.684967 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 06:14:55.685067 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 06:14:55.685210 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":86,"threads_started":1}
I20260926 06:14:55.705701 3106 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:14:55.706475 3106 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:14:55.719359 3106 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0b89bc3234c40d3a2ca000bea7742ae: Generated new TSK 0
I20260926 06:14:55.719604 3106 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab.
Entry for principal kudu/127.2.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab.
I20260926 06:14:55.752909 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.193:0
--local_ip_for_outbound_sockets=127.2.110.193
--webserver_interface=127.2.110.193
--webserver_port=0
--tserver_master_addrs=127.2.110.254:38819
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab
--principal=kudu/127.2.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:45547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:14:55.854210 3123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:14:55.854521 3123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:14:55.854568 3123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:14:55.856074 3123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:14:55.856153 3123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:14:55.856297 3123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.193
I20260926 06:14:55.858283 3123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:45547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab
--principal=kudu/127.2.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.110.193
--webserver_port=0
--tserver_master_addrs=127.2.110.254:38819
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.193
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:14:55.858598 3123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:14:55.858856 3123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:14:55.864548 3128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:14:55.864562 3131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:14:55.864658 3123 server_base.cc:1034] running on GCE node
W20260926 06:14:55.864564 3129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:14:55.864975 3123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:14:55.865245 3123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:14:55.866413 3123 hybrid_clock.cc:648] HybridClock initialized: now 1790403295866392 us; error 39 us; skew 500 ppm
Sep 26 06:14:55 dist-test-slave-hndd krb5kdc[3016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403295, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:14:55.870406 3123 init.cc:377] Logged in from keytab as kudu/127.2.110.193@KRBTEST.COM (short username kudu)
I20260926 06:14:55.870996 3123 webserver.cc:460] Webserver started at http://127.2.110.193:39185/ using document root <none> and password file <none>
I20260926 06:14:55.871310 3123 fs_manager.cc:359] Metadata directory not provided
I20260926 06:14:55.871369 3123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:14:55.871548 3123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:14:55.872831 3123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/ts-0/data/instance:
uuid: "399885d48c014147b59f3a4e18c080c6"
format_stamp: "Formatted at 2026-09-26 06:14:55 on dist-test-slave-hndd"
server_key: "6edb4145694855a150bfd4c07f1e9427"
server_key_iv: "daca5bfc8cc7d9c7db1b3e2f1512530b"
server_key_version: "encryptionkey@0"
I20260926 06:14:55.873274 3123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/ts-0/wal/instance:
uuid: "399885d48c014147b59f3a4e18c080c6"
format_stamp: "Formatted at 2026-09-26 06:14:55 on dist-test-slave-hndd"
server_key: "6edb4145694855a150bfd4c07f1e9427"
server_key_iv: "daca5bfc8cc7d9c7db1b3e2f1512530b"
server_key_version: "encryptionkey@0"
I20260926 06:14:55.875461 3123 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 06:14:55.876617 3138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:14:55.876879 3123 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:14:55.876996 3123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/ts-0/wal
uuid: "399885d48c014147b59f3a4e18c080c6"
format_stamp: "Formatted at 2026-09-26 06:14:55 on dist-test-slave-hndd"
server_key: "6edb4145694855a150bfd4c07f1e9427"
server_key_iv: "daca5bfc8cc7d9c7db1b3e2f1512530b"
server_key_version: "encryptionkey@0"
I20260926 06:14:55.877084 3123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:14:55.908346 3123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:14:55.910872 3123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:14:55.911032 3123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:14:55.911531 3123 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:14:55.911981 3123 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:14:55.912024 3123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:14:55.912050 3123 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:14:55.912099 3123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:14:55.918394 3123 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.193:38909
I20260926 06:14:55.918479 3251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.193:38909 every 8 connection(s)
I20260926 06:14:55.918844 3123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
I20260926 06:14:55.919013 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 3123
I20260926 06:14:55.919137 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/ts-0/wal/instance
I20260926 06:14:55.919353 2491 external_mini_cluster.cc:1428] Setting key 44f16b6f43627f8b7a95feea5534be0d
Sep 26 06:14:55 dist-test-slave-hndd krb5kdc[3016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403295, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.193@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.110.194@KRBTEST.COM; defaulting to no policy
I20260926 06:14:55.930711 3101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:14:55.919760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:54345 (local address 127.2.110.254:38819)
0926 06:14:55.919804 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 06:14:55.919806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:14:55.920113 (+ 307us) server_negotiation.cc:407] Connection header received
0926 06:14:55.921100 (+ 987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:14:55.921101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:14:55.921114 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:14:55.921173 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:14:55.921909 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:14:55.922432 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:14:55.923491 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:14:55.923725 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:14:55.923814 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:14:55.926632 (+ 2818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:14:55.926635 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:14:55.926635 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:14:55.926647 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:14:55.928668 (+ 2021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:14:55.929510 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:14:55.929512 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:14:55.929513 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:14:55.929639 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:14:55.929994 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:14:55.929995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:14:55.929996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:14:55.930141 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:14:55.930223 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 06:14:55.930519 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 06:14:55.930615 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 06:14:55.930922 3254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:14:55.919810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:38819 (local address 127.2.110.193:54345)
0926 06:14:55.919965 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 06:14:55.920003 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 06:14:55.920987 (+ 984us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:14:55.921252 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:14:55.921256 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:14:55.921610 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:14:55.921809 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:14:55.921814 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:14:55.922533 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:14:55.922535 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:14:55.923373 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:14:55.923375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:14:55.923825 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:14:55.923826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:14:55.923966 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:14:55.924564 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:14:55.924572 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:14:55.926475 (+ 1903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:14:55.928827 (+ 2352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:14:55.928838 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:14:55.928841 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:14:55.929390 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:14:55.929761 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:14:55.929762 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:14:55.929762 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:14:55.929914 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:14:55.930215 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:14:55.930226 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:14:55.930378 (+ 152us) client_negotiation.cc:769] Sending connection context
0926 06:14:55.930482 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 06:14:55.930660 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":88,"threads_started":1}
I20260926 06:14:55.931541 3252 heartbeater.cc:347] Connected to a master server at 127.2.110.254:38819
I20260926 06:14:55.931659 3252 heartbeater.cc:464] Registering TS with master...
I20260926 06:14:55.931850 3252 heartbeater.cc:511] Master 127.2.110.254:38819 requested a full tablet report, sending...
I20260926 06:14:55.932524 3061 ts_manager.cc:194] Registered new tserver with Master: 399885d48c014147b59f3a4e18c080c6 (127.2.110.193:38909)
I20260926 06:14:55.933463 3061 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.193@KRBTEST.COM'} at 127.2.110.193:54345
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.194@KRBTEST.COM" created.
I20260926 06:14:56.934732 3252 heartbeater.cc:503] Master 127.2.110.254:38819 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab.
Entry for principal kudu/127.2.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab.
W20260926 06:14:57.532064 2491 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.2.110.194: real 1.612s user 0.000s sys 0.003s
I20260926 06:14:57.532464 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.194:0
--local_ip_for_outbound_sockets=127.2.110.194
--webserver_interface=127.2.110.194
--webserver_port=0
--tserver_master_addrs=127.2.110.254:38819
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab
--principal=kudu/127.2.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:45547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:14:57.637391 3257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:14:57.637645 3257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:14:57.637691 3257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:14:57.639052 3257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:14:57.639114 3257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:14:57.639163 3257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.194
I20260926 06:14:57.640870 3257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:45547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab
--principal=kudu/127.2.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.110.194
--webserver_port=0
--tserver_master_addrs=127.2.110.254:38819
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.194
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:14:57.641178 3257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:14:57.641456 3257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:14:57.646737 3263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:14:57.646759 3265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:14:57.646746 3262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:14:57.647015 3257 server_base.cc:1034] running on GCE node
I20260926 06:14:57.647300 3257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:14:57.647526 3257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:14:57.648689 3257 hybrid_clock.cc:648] HybridClock initialized: now 1790403297648673 us; error 32 us; skew 500 ppm
Sep 26 06:14:57 dist-test-slave-hndd krb5kdc[3016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403297, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:14:57.651835 3257 init.cc:377] Logged in from keytab as kudu/127.2.110.194@KRBTEST.COM (short username kudu)
I20260926 06:14:57.652369 3257 webserver.cc:460] Webserver started at http://127.2.110.194:35179/ using document root <none> and password file <none>
I20260926 06:14:57.652590 3257 fs_manager.cc:359] Metadata directory not provided
I20260926 06:14:57.652634 3257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:14:57.652735 3257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:14:57.654019 3257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/ts-1/data/instance:
uuid: "67d86f551cbc464cb3be3b10063e7934"
format_stamp: "Formatted at 2026-09-26 06:14:57 on dist-test-slave-hndd"
server_key: "0187592086001d46b49d7315e659e849"
server_key_iv: "14e3b16518ea44ee2e1a7902847f11bf"
server_key_version: "encryptionkey@0"
I20260926 06:14:57.654368 3257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/ts-1/wal/instance:
uuid: "67d86f551cbc464cb3be3b10063e7934"
format_stamp: "Formatted at 2026-09-26 06:14:57 on dist-test-slave-hndd"
server_key: "0187592086001d46b49d7315e659e849"
server_key_iv: "14e3b16518ea44ee2e1a7902847f11bf"
server_key_version: "encryptionkey@0"
I20260926 06:14:57.655972 3257 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:14:57.656842 3272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:14:57.657167 3257 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:14:57.657258 3257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/ts-1/wal
uuid: "67d86f551cbc464cb3be3b10063e7934"
format_stamp: "Formatted at 2026-09-26 06:14:57 on dist-test-slave-hndd"
server_key: "0187592086001d46b49d7315e659e849"
server_key_iv: "14e3b16518ea44ee2e1a7902847f11bf"
server_key_version: "encryptionkey@0"
I20260926 06:14:57.657403 3257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:14:57.675482 3257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:14:57.678553 3257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:14:57.678748 3257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:14:57.679088 3257 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:14:57.679472 3257 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:14:57.679510 3257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:14:57.679535 3257 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:14:57.679548 3257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:14:57.686113 3257 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.194:39569
I20260926 06:14:57.686218 3385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.194:39569 every 8 connection(s)
I20260926 06:14:57.686594 3257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
I20260926 06:14:57.689560 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 3257
I20260926 06:14:57.689688 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/ts-1/wal/instance
I20260926 06:14:57.689858 2491 external_mini_cluster.cc:1428] Setting key 2bad730aac2a376c9eb7593fcc73c263
Sep 26 06:14:57 dist-test-slave-hndd krb5kdc[3016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403297, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.194@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.110.195@KRBTEST.COM; defaulting to no policy
I20260926 06:14:57.698031 3388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:14:57.687416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:34001 (local address 127.2.110.254:38819)
0926 06:14:57.687590 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 06:14:57.687592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:14:57.687746 (+ 154us) server_negotiation.cc:407] Connection header received
0926 06:14:57.689235 (+ 1489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:14:57.689237 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:14:57.689259 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:14:57.689327 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:14:57.690380 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:14:57.690927 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:14:57.692148 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:14:57.692398 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:14:57.692492 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:14:57.694929 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:14:57.694931 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:14:57.694931 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:14:57.694939 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:14:57.696480 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:14:57.696924 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:14:57.696925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:14:57.696926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:14:57.696989 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:14:57.697239 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:14:57.697240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:14:57.697240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:14:57.697445 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:14:57.697521 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 06:14:57.697835 (+ 314us) server_negotiation.cc:299] Negotiation successful
0926 06:14:57.697927 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":81,"threads_started":1}
I20260926 06:14:57.698231 3389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:14:57.687412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:38819 (local address 127.2.110.194:34001)
0926 06:14:57.687608 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 06:14:57.687657 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 06:14:57.688970 (+ 1313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:14:57.689472 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:14:57.689477 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:14:57.689858 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:14:57.690260 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:14:57.690267 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:14:57.691024 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:14:57.691025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:14:57.692036 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:14:57.692039 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:14:57.692503 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:14:57.692504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:14:57.692647 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:14:57.693256 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:14:57.693266 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:14:57.694819 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:14:57.696591 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:14:57.696600 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:14:57.696602 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:14:57.696841 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:14:57.697054 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:14:57.697055 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:14:57.697055 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:14:57.697168 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:14:57.697509 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:14:57.697515 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:14:57.697671 (+ 156us) client_negotiation.cc:769] Sending connection context
0926 06:14:57.697799 (+ 128us) client_negotiation.cc:240] Negotiation successful
0926 06:14:57.697981 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":102,"threads_started":1}
I20260926 06:14:57.698745 3386 heartbeater.cc:347] Connected to a master server at 127.2.110.254:38819
I20260926 06:14:57.698863 3386 heartbeater.cc:464] Registering TS with master...
I20260926 06:14:57.699052 3386 heartbeater.cc:511] Master 127.2.110.254:38819 requested a full tablet report, sending...
I20260926 06:14:57.699456 3061 ts_manager.cc:194] Registered new tserver with Master: 67d86f551cbc464cb3be3b10063e7934 (127.2.110.194:39569)
I20260926 06:14:57.699848 3061 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.194@KRBTEST.COM'} at 127.2.110.194:34001
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.195@KRBTEST.COM" created.
I20260926 06:14:58.700749 3386 heartbeater.cc:503] Master 127.2.110.254:38819 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab.
Entry for principal kudu/127.2.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab.
W20260926 06:14:59.043432 2491 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.2.110.195: real 1.353s user 0.000s sys 0.003s
I20260926 06:14:59.043818 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.195:0
--local_ip_for_outbound_sockets=127.2.110.195
--webserver_interface=127.2.110.195
--webserver_port=0
--tserver_master_addrs=127.2.110.254:38819
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab
--principal=kudu/127.2.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:45547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:14:59.134728 3392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:14:59.134999 3392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:14:59.135039 3392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:14:59.136308 3392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:14:59.136376 3392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:14:59.136425 3392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.195
I20260926 06:14:59.138103 3392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:45547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab
--principal=kudu/127.2.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.110.195
--webserver_port=0
--tserver_master_addrs=127.2.110.254:38819
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.195
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:14:59.138414 3392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:14:59.138656 3392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:14:59.144079 3397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:14:59.144011 3398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:14:59.144099 3392 server_base.cc:1034] running on GCE node
W20260926 06:14:59.144008 3400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:14:59.144382 3392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:14:59.144644 3392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:14:59.145840 3392 hybrid_clock.cc:648] HybridClock initialized: now 1790403299145806 us; error 52 us; skew 500 ppm
Sep 26 06:14:59 dist-test-slave-hndd krb5kdc[3016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403299, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:14:59.155359 3392 init.cc:377] Logged in from keytab as kudu/127.2.110.195@KRBTEST.COM (short username kudu)
I20260926 06:14:59.155848 3392 webserver.cc:460] Webserver started at http://127.2.110.195:43745/ using document root <none> and password file <none>
I20260926 06:14:59.156083 3392 fs_manager.cc:359] Metadata directory not provided
I20260926 06:14:59.156152 3392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:14:59.156306 3392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:14:59.157498 3392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/ts-2/data/instance:
uuid: "02fcaedf69fe4ab7966328cc2c8abd5f"
format_stamp: "Formatted at 2026-09-26 06:14:59 on dist-test-slave-hndd"
server_key: "84a71c571d19fe75305ee1fa6304df3e"
server_key_iv: "a521b2426949376c502052db7c2b2de7"
server_key_version: "encryptionkey@0"
I20260926 06:14:59.157866 3392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/ts-2/wal/instance:
uuid: "02fcaedf69fe4ab7966328cc2c8abd5f"
format_stamp: "Formatted at 2026-09-26 06:14:59 on dist-test-slave-hndd"
server_key: "84a71c571d19fe75305ee1fa6304df3e"
server_key_iv: "a521b2426949376c502052db7c2b2de7"
server_key_version: "encryptionkey@0"
I20260926 06:14:59.159369 3392 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:14:59.160264 3407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:14:59.160475 3392 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:14:59.160578 3392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/ts-2/wal
uuid: "02fcaedf69fe4ab7966328cc2c8abd5f"
format_stamp: "Formatted at 2026-09-26 06:14:59 on dist-test-slave-hndd"
server_key: "84a71c571d19fe75305ee1fa6304df3e"
server_key_iv: "a521b2426949376c502052db7c2b2de7"
server_key_version: "encryptionkey@0"
I20260926 06:14:59.160660 3392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:14:59.207504 3392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:14:59.210417 3392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:14:59.210587 3392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:14:59.210870 3392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:14:59.211230 3392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:14:59.211277 3392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:14:59.211313 3392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:14:59.211336 3392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:14:59.217936 3392 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.195:45571
I20260926 06:14:59.218078 3520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.195:45571 every 8 connection(s)
I20260926 06:14:59.218420 3392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
I20260926 06:14:59.219907 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 3392
I20260926 06:14:59.220037 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403292369990-2491-0/minicluster-data/ts-2/wal/instance
I20260926 06:14:59.220235 2491 external_mini_cluster.cc:1428] Setting key ae8d367d3733d45f1a74cbd0492ef514
Sep 26 06:14:59 dist-test-slave-hndd krb5kdc[3016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403299, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.195@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:14:59.229305 3523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:14:59.219283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:48547 (local address 127.2.110.254:38819)
0926 06:14:59.219419 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 06:14:59.219421 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:14:59.219595 (+ 174us) server_negotiation.cc:407] Connection header received
0926 06:14:59.221039 (+ 1444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:14:59.221042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:14:59.221072 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:14:59.221194 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:14:59.222028 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:14:59.222534 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:14:59.223604 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:14:59.223811 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:14:59.223869 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:14:59.226089 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:14:59.226090 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:14:59.226090 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:14:59.226097 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:14:59.227870 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:14:59.228383 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:14:59.228384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:14:59.228384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:14:59.228455 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:14:59.228678 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:14:59.228679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:14:59.228679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:14:59.228798 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:14:59.228860 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 06:14:59.229143 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 06:14:59.229229 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":83,"threads_started":1}
I20260926 06:14:59.229557 3524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:14:59.219249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:38819 (local address 127.2.110.195:48547)
0926 06:14:59.219460 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 06:14:59.219504 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 06:14:59.220772 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:14:59.221295 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:14:59.221301 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:14:59.221730 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:14:59.221944 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:14:59.221948 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:14:59.222619 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:14:59.222620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:14:59.223524 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:14:59.223526 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:14:59.223883 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:14:59.223884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:14:59.223988 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:14:59.224555 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:14:59.224563 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:14:59.225999 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:14:59.228060 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:14:59.228066 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:14:59.228069 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:14:59.228312 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:14:59.228513 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:14:59.228514 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:14:59.228514 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:14:59.228615 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:14:59.228853 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:14:59.228857 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:14:59.228984 (+ 127us) client_negotiation.cc:769] Sending connection context
0926 06:14:59.229119 (+ 135us) client_negotiation.cc:240] Negotiation successful
0926 06:14:59.229254 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":114,"threads_started":1}
I20260926 06:14:59.230306 3521 heartbeater.cc:347] Connected to a master server at 127.2.110.254:38819
I20260926 06:14:59.230444 3521 heartbeater.cc:464] Registering TS with master...
I20260926 06:14:59.230655 3521 heartbeater.cc:511] Master 127.2.110.254:38819 requested a full tablet report, sending...
I20260926 06:14:59.231065 3061 ts_manager.cc:194] Registered new tserver with Master: 02fcaedf69fe4ab7966328cc2c8abd5f (127.2.110.195:45571)
I20260926 06:14:59.231444 3061 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.195@KRBTEST.COM'} at 127.2.110.195:48547
I20260926 06:14:59.232587 2491 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 06:14:59.241113 3523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:14:59.234368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44050 (local address 127.2.110.254:38819)
0926 06:14:59.234423 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 06:14:59.234425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:14:59.234587 (+ 162us) server_negotiation.cc:407] Connection header received
0926 06:14:59.234663 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:14:59.234664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:14:59.234674 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:14:59.234725 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:14:59.235324 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:14:59.235819 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:14:59.236552 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:14:59.236763 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:14:59.236846 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:14:59.238276 (+ 1430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:14:59.238278 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:14:59.238279 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:14:59.238285 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:14:59.239799 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:14:59.240237 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:14:59.240238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:14:59.240239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:14:59.240295 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:14:59.240503 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:14:59.240504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:14:59.240504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:14:59.240661 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:14:59.240770 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 06:14:59.240912 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 06:14:59.241005 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 06:14:59.243402 3061 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:14:59.245090 3061 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:14:59.252923 3536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:14:59.248396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42442 (local address 127.2.110.193:38909)
0926 06:14:59.248552 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 06:14:59.248557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:14:59.248569 (+ 12us) server_negotiation.cc:407] Connection header received
0926 06:14:59.248627 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:14:59.248629 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:14:59.248648 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:14:59.248745 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:14:59.249227 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:14:59.250505 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:14:59.251766 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:14:59.252471 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:14:59.252527 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:14:59.252574 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 06:14:59.252771 (+ 197us) server_negotiation.cc:299] Negotiation successful
0926 06:14:59.252824 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":64,"threads_started":1}
I20260926 06:14:59.252923 3534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:14:59.248091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.193:38909 (local address 127.0.0.1:42442)
0926 06:14:59.248281 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 06:14:59.248320 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 06:14:59.248390 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:14:59.249098 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:14:59.249098 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:14:59.249100 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:14:59.249130 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:14:59.249132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:14:59.250603 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:14:59.250604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:14:59.251683 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:14:59.251684 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:14:59.252607 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:14:59.252608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:14:59.252674 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:14:59.252677 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:14:59.252754 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 06:14:59.252839 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":64,"thread_start_us":113,"threads_started":1}
I20260926 06:14:59.253101 3537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:14:59.248405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52862 (local address 127.2.110.195:45571)
0926 06:14:59.248872 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0926 06:14:59.248876 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:14:59.248884 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:14:59.248915 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:14:59.248916 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:14:59.248928 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:14:59.248993 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:14:59.249200 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:14:59.250103 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:14:59.251635 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:14:59.252394 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:14:59.252478 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:14:59.252544 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 06:14:59.252972 (+ 428us) server_negotiation.cc:299] Negotiation successful
0926 06:14:59.253025 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":83,"threads_started":1}
I20260926 06:14:59.252933 3535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:14:59.248091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.195:45571 (local address 127.0.0.1:52862)
0926 06:14:59.248500 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0926 06:14:59.248509 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 06:14:59.248549 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:14:59.249023 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:14:59.249026 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:14:59.249029 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:14:59.249116 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:14:59.249118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:14:59.250211 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:14:59.250211 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:14:59.251445 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:14:59.251448 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:14:59.252552 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:14:59.252553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:14:59.252645 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:14:59.252660 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 06:14:59.252746 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 06:14:59.252829 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":2,"thread_start_us":114,"threads_started":1}
I20260926 06:14:59.253291 3538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:14:59.248509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.194:39569 (local address 127.0.0.1:33242)
0926 06:14:59.248655 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 06:14:59.248667 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:14:59.248719 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:14:59.249054 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:14:59.249055 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:14:59.249063 (+ 8us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:14:59.249122 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:14:59.249124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:14:59.250618 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:14:59.250619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:14:59.251774 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:14:59.251777 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:14:59.252931 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:14:59.252932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:14:59.252990 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:14:59.252994 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:14:59.253042 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 06:14:59.253084 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":1,"thread_start_us":86,"threads_started":1}
I20260926 06:14:59.253629 3539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:14:59.248748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33242 (local address 127.2.110.194:39569)
0926 06:14:59.248882 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 06:14:59.248886 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:14:59.248896 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:14:59.248934 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:14:59.248935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:14:59.248947 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:14:59.249039 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:14:59.249228 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:14:59.250509 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:14:59.251872 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:14:59.252776 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:14:59.252838 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:14:59.252896 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 06:14:59.253204 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 06:14:59.253264 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":76,"threads_started":1}
I20260926 06:14:59.254240 3447 tablet_service.cc:1467] Processing CreateTablet for tablet 2d10c7194870412db75977141addcf21 (DEFAULT_TABLE table=test-table [id=3b44caa3615b45b2a3ea757e5553d7bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:14:59.254238 3320 tablet_service.cc:1467] Processing CreateTablet for tablet 2d10c7194870412db75977141addcf21 (DEFAULT_TABLE table=test-table [id=3b44caa3615b45b2a3ea757e5553d7bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:14:59.254245 3186 tablet_service.cc:1467] Processing CreateTablet for tablet 2d10c7194870412db75977141addcf21 (DEFAULT_TABLE table=test-table [id=3b44caa3615b45b2a3ea757e5553d7bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:14:59.254638 3447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d10c7194870412db75977141addcf21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:14:59.254638 3320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d10c7194870412db75977141addcf21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:14:59.254644 3186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d10c7194870412db75977141addcf21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:14:59.257988 3540 tablet_bootstrap.cc:492] T 2d10c7194870412db75977141addcf21 P 399885d48c014147b59f3a4e18c080c6: Bootstrap starting.
I20260926 06:14:59.258607 3541 tablet_bootstrap.cc:492] T 2d10c7194870412db75977141addcf21 P 67d86f551cbc464cb3be3b10063e7934: Bootstrap starting.
I20260926 06:14:59.258656 3542 tablet_bootstrap.cc:492] T 2d10c7194870412db75977141addcf21 P 02fcaedf69fe4ab7966328cc2c8abd5f: Bootstrap starting.
I20260926 06:14:59.258862 3540 tablet_bootstrap.cc:654] T 2d10c7194870412db75977141addcf21 P 399885d48c014147b59f3a4e18c080c6: Neither blocks nor log segments found. Creating new log.
I20260926 06:14:59.259225 3540 log.cc:824] T 2d10c7194870412db75977141addcf21 P 399885d48c014147b59f3a4e18c080c6: Log is configured to *not* fsync() on all Append() calls
I20260926 06:14:59.259326 3541 tablet_bootstrap.cc:654] T 2d10c7194870412db75977141addcf21 P 67d86f551cbc464cb3be3b10063e7934: Neither blocks nor log segments found. Creating new log.
I20260926 06:14:59.259357 3542 tablet_bootstrap.cc:654] T 2d10c7194870412db75977141addcf21 P 02fcaedf69fe4ab7966328cc2c8abd5f: Neither blocks nor log segments found. Creating new log.
I20260926 06:14:59.259621 3541 log.cc:824] T 2d10c7194870412db75977141addcf21 P 67d86f551cbc464cb3be3b10063e7934: Log is configured to *not* fsync() on all Append() calls
I20260926 06:14:59.259945 3542 log.cc:824] T 2d10c7194870412db75977141addcf21 P 02fcaedf69fe4ab7966328cc2c8abd5f: Log is configured to *not* fsync() on all Append() calls
I20260926 06:14:59.260423 3540 tablet_bootstrap.cc:492] T 2d10c7194870412db75977141addcf21 P 399885d48c014147b59f3a4e18c080c6: No bootstrap required, opened a new log
I20260926 06:14:59.260433 3541 tablet_bootstrap.cc:492] T 2d10c7194870412db75977141addcf21 P 67d86f551cbc464cb3be3b10063e7934: No bootstrap required, opened a new log
I20260926 06:14:59.260496 3541 ts_tablet_manager.cc:1397] T 2d10c7194870412db75977141addcf21 P 67d86f551cbc464cb3be3b10063e7934: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:14:59.260480 3540 ts_tablet_manager.cc:1397] T 2d10c7194870412db75977141addcf21 P 399885d48c014147b59f3a4e18c080c6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 06:14:59.260649 3542 tablet_bootstrap.cc:492] T 2d10c7194870412db75977141addcf21 P 02fcaedf69fe4ab7966328cc2c8abd5f: No bootstrap required, opened a new log
I20260926 06:14:59.260697 3542 ts_tablet_manager.cc:1397] T 2d10c7194870412db75977141addcf21 P 02fcaedf69fe4ab7966328cc2c8abd5f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:14:59.262872 3540 raft_consensus.cc:348] T 2d10c7194870412db75977141addcf21 P 399885d48c014147b59f3a4e18c080c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d86f551cbc464cb3be3b10063e7934" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 39569 } } peers { permanent_uuid: "399885d48c014147b59f3a4e18c080c6" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 38909 } } peers { permanent_uuid: "02fcaedf69fe4ab7966328cc2c8abd5f" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 45571 } }
I20260926 06:14:59.263015 3540 raft_consensus.cc:374] T 2d10c7194870412db75977141addcf21 P 399885d48c014147b59f3a4e18c080c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:14:59.263036 3540 raft_consensus.cc:729] T 2d10c7194870412db75977141addcf21 P 399885d48c014147b59f3a4e18c080c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 399885d48c014147b59f3a4e18c080c6, State: Initialized, Role: FOLLOWER
I20260926 06:14:59.263175 3540 consensus_queue.cc:260] T 2d10c7194870412db75977141addcf21 P 399885d48c014147b59f3a4e18c080c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d86f551cbc464cb3be3b10063e7934" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 39569 } } peers { permanent_uuid: "399885d48c014147b59f3a4e18c080c6" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 38909 } } peers { permanent_uuid: "02fcaedf69fe4ab7966328cc2c8abd5f" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 45571 } }
I20260926 06:14:59.263073 3542 raft_consensus.cc:348] T 2d10c7194870412db75977141addcf21 P 02fcaedf69fe4ab7966328cc2c8abd5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d86f551cbc464cb3be3b10063e7934" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 39569 } } peers { permanent_uuid: "399885d48c014147b59f3a4e18c080c6" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 38909 } } peers { permanent_uuid: "02fcaedf69fe4ab7966328cc2c8abd5f" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 45571 } }
I20260926 06:14:59.263271 3542 raft_consensus.cc:374] T 2d10c7194870412db75977141addcf21 P 02fcaedf69fe4ab7966328cc2c8abd5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:14:59.263305 3542 raft_consensus.cc:729] T 2d10c7194870412db75977141addcf21 P 02fcaedf69fe4ab7966328cc2c8abd5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02fcaedf69fe4ab7966328cc2c8abd5f, State: Initialized, Role: FOLLOWER
I20260926 06:14:59.263477 3540 ts_tablet_manager.cc:1428] T 2d10c7194870412db75977141addcf21 P 399885d48c014147b59f3a4e18c080c6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 06:14:59.263450 3542 consensus_queue.cc:260] T 2d10c7194870412db75977141addcf21 P 02fcaedf69fe4ab7966328cc2c8abd5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d86f551cbc464cb3be3b10063e7934" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 39569 } } peers { permanent_uuid: "399885d48c014147b59f3a4e18c080c6" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 38909 } } peers { permanent_uuid: "02fcaedf69fe4ab7966328cc2c8abd5f" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 45571 } }
I20260926 06:14:59.263712 3521 heartbeater.cc:503] Master 127.2.110.254:38819 was elected leader, sending a full tablet report...
I20260926 06:14:59.263707 3542 ts_tablet_manager.cc:1428] T 2d10c7194870412db75977141addcf21 P 02fcaedf69fe4ab7966328cc2c8abd5f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 06:14:59.263852 3541 raft_consensus.cc:348] T 2d10c7194870412db75977141addcf21 P 67d86f551cbc464cb3be3b10063e7934 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d86f551cbc464cb3be3b10063e7934" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 39569 } } peers { permanent_uuid: "399885d48c014147b59f3a4e18c080c6" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 38909 } } peers { permanent_uuid: "02fcaedf69fe4ab7966328cc2c8abd5f" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 45571 } }
I20260926 06:14:59.263995 3541 raft_consensus.cc:374] T 2d10c7194870412db75977141addcf21 P 67d86f551cbc464cb3be3b10063e7934 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:14:59.264031 3541 raft_consensus.cc:729] T 2d10c7194870412db75977141addcf21 P 67d86f551cbc464cb3be3b10063e7934 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67d86f551cbc464cb3be3b10063e7934, State: Initialized, Role: FOLLOWER
I20260926 06:14:59.264299 3541 consensus_queue.cc:260] T 2d10c7194870412db75977141addcf21 P 67d86f551cbc464cb3be3b10063e7934 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d86f551cbc464cb3be3b10063e7934" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 39569 } } peers { permanent_uuid: "399885d48c014147b59f3a4e18c080c6" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 38909 } } peers { permanent_uuid: "02fcaedf69fe4ab7966328cc2c8abd5f" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 45571 } }
I20260926 06:14:59.264583 3541 ts_tablet_manager.cc:1428] T 2d10c7194870412db75977141addcf21 P 67d86f551cbc464cb3be3b10063e7934: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260926 06:14:59.317461 3546 raft_consensus.cc:482] T 2d10c7194870412db75977141addcf21 P 399885d48c014147b59f3a4e18c080c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:14:59.317634 3546 raft_consensus.cc:504] T 2d10c7194870412db75977141addcf21 P 399885d48c014147b59f3a4e18c080c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d86f551cbc464cb3be3b10063e7934" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 39569 } } peers { permanent_uuid: "399885d48c014147b59f3a4e18c080c6" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 38909 } } peers { permanent_uuid: "02fcaedf69fe4ab7966328cc2c8abd5f" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 45571 } }
I20260926 06:14:59.318043 3546 leader_election.cc:288] T 2d10c7194870412db75977141addcf21 P 399885d48c014147b59f3a4e18c080c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67d86f551cbc464cb3be3b10063e7934 (127.2.110.194:39569), 02fcaedf69fe4ab7966328cc2c8abd5f (127.2.110.195:45571)
I20260926 06:14:59.321563 3537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:14:59.318684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:58039 (local address 127.2.110.195:45571)
0926 06:14:59.318779 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0926 06:14:59.318781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:14:59.318853 (+ 72us) server_negotiation.cc:407] Connection header received
0926 06:14:59.318892 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:14:59.318893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:14:59.318907 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:14:59.318972 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:14:59.319179 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:14:59.319691 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:14:59.320789 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:14:59.321151 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:14:59.321248 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:14:59.321262 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0926 06:14:59.321439 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 06:14:59.321482 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 06:14:59.321585 3549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:14:59.318300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.195:45571 (local address 127.2.110.193:58039)
0926 06:14:59.318761 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0926 06:14:59.318772 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:14:59.318846 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:14:59.318995 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:14:59.318996 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:14:59.318999 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:14:59.319096 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:14:59.319099 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:14:59.319778 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:14:59.319780 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:14:59.320663 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:14:59.320665 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:14:59.321258 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:14:59.321259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:14:59.321362 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:14:59.321368 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 06:14:59.321431 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 06:14:59.321480 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"mutex_wait_us":46,"thread_start_us":126,"threads_started":1}
I20260926 06:14:59.321653 3539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:14:59.318605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:48261 (local address 127.2.110.194:39569)
0926 06:14:59.318729 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 06:14:59.318732 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:14:59.318997 (+ 265us) server_negotiation.cc:407] Connection header received
0926 06:14:59.319028 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:14:59.319029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:14:59.319039 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:14:59.319093 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:14:59.319263 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:14:59.319760 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:14:59.320753 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:14:59.321134 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:14:59.321383 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:14:59.321401 (+ 18us) server_negotiation.cc:1035] Waiting for connection context
0926 06:14:59.321539 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 06:14:59.321578 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 06:14:59.321900 3550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:14:59.318307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.194:39569 (local address 127.2.110.193:48261)
0926 06:14:59.318876 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0926 06:14:59.318887 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:14:59.318959 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:14:59.319106 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:14:59.319106 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:14:59.319108 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:14:59.319190 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:14:59.319192 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:14:59.319845 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:14:59.319846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:14:59.320656 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:14:59.320657 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:14:59.321384 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:14:59.321385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:14:59.321464 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:14:59.321468 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:14:59.321526 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 06:14:59.321562 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":125,"threads_started":1}
I20260926 06:14:59.322010 3473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d10c7194870412db75977141addcf21" candidate_uuid: "399885d48c014147b59f3a4e18c080c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02fcaedf69fe4ab7966328cc2c8abd5f" is_pre_election: true
I20260926 06:14:59.322192 3473 raft_consensus.cc:2454] T 2d10c7194870412db75977141addcf21 P 02fcaedf69fe4ab7966328cc2c8abd5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 399885d48c014147b59f3a4e18c080c6 in term 0.
I20260926 06:14:59.322247 3340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d10c7194870412db75977141addcf21" candidate_uuid: "399885d48c014147b59f3a4e18c080c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67d86f551cbc464cb3be3b10063e7934" is_pre_election: true
I20260926 06:14:59.322394 3340 raft_consensus.cc:2454] T 2d10c7194870412db75977141addcf21 P 67d86f551cbc464cb3be3b10063e7934 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 399885d48c014147b59f3a4e18c080c6 in term 0.
I20260926 06:14:59.322453 3142 leader_election.cc:302] T 2d10c7194870412db75977141addcf21 P 399885d48c014147b59f3a4e18c080c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fcaedf69fe4ab7966328cc2c8abd5f, 399885d48c014147b59f3a4e18c080c6; no voters:
I20260926 06:14:59.322645 3546 raft_consensus.cc:2781] T 2d10c7194870412db75977141addcf21 P 399885d48c014147b59f3a4e18c080c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:14:59.322701 3546 raft_consensus.cc:482] T 2d10c7194870412db75977141addcf21 P 399885d48c014147b59f3a4e18c080c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:14:59.322718 3546 raft_consensus.cc:3037] T 2d10c7194870412db75977141addcf21 P 399885d48c014147b59f3a4e18c080c6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:14:59.405673 3546 raft_consensus.cc:504] T 2d10c7194870412db75977141addcf21 P 399885d48c014147b59f3a4e18c080c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d86f551cbc464cb3be3b10063e7934" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 39569 } } peers { permanent_uuid: "399885d48c014147b59f3a4e18c080c6" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 38909 } } peers { permanent_uuid: "02fcaedf69fe4ab7966328cc2c8abd5f" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 45571 } }
I20260926 06:14:59.405936 3546 leader_election.cc:288] T 2d10c7194870412db75977141addcf21 P 399885d48c014147b59f3a4e18c080c6 [CANDIDATE]: Term 1 election: Requested vote from peers 67d86f551cbc464cb3be3b10063e7934 (127.2.110.194:39569), 02fcaedf69fe4ab7966328cc2c8abd5f (127.2.110.195:45571)
I20260926 06:14:59.406227 3340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d10c7194870412db75977141addcf21" candidate_uuid: "399885d48c014147b59f3a4e18c080c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67d86f551cbc464cb3be3b10063e7934"
I20260926 06:14:59.406229 3473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d10c7194870412db75977141addcf21" candidate_uuid: "399885d48c014147b59f3a4e18c080c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02fcaedf69fe4ab7966328cc2c8abd5f"
I20260926 06:14:59.406339 3340 raft_consensus.cc:3037] T 2d10c7194870412db75977141addcf21 P 67d86f551cbc464cb3be3b10063e7934 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:14:59.406369 3473 raft_consensus.cc:3037] T 2d10c7194870412db75977141addcf21 P 02fcaedf69fe4ab7966328cc2c8abd5f [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:14:59.407322 3340 raft_consensus.cc:2454] T 2d10c7194870412db75977141addcf21 P 67d86f551cbc464cb3be3b10063e7934 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 399885d48c014147b59f3a4e18c080c6 in term 1.
I20260926 06:14:59.407490 3473 raft_consensus.cc:2454] T 2d10c7194870412db75977141addcf21 P 02fcaedf69fe4ab7966328cc2c8abd5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 399885d48c014147b59f3a4e18c080c6 in term 1.
I20260926 06:14:59.407582 3140 leader_election.cc:302] T 2d10c7194870412db75977141addcf21 P 399885d48c014147b59f3a4e18c080c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 399885d48c014147b59f3a4e18c080c6, 67d86f551cbc464cb3be3b10063e7934; no voters:
I20260926 06:14:59.407691 3546 raft_consensus.cc:2781] T 2d10c7194870412db75977141addcf21 P 399885d48c014147b59f3a4e18c080c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:14:59.407872 3546 raft_consensus.cc:686] T 2d10c7194870412db75977141addcf21 P 399885d48c014147b59f3a4e18c080c6 [term 1 LEADER]: Becoming Leader. State: Replica: 399885d48c014147b59f3a4e18c080c6, State: Running, Role: LEADER
I20260926 06:14:59.407999 3546 consensus_queue.cc:237] T 2d10c7194870412db75977141addcf21 P 399885d48c014147b59f3a4e18c080c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d86f551cbc464cb3be3b10063e7934" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 39569 } } peers { permanent_uuid: "399885d48c014147b59f3a4e18c080c6" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 38909 } } peers { permanent_uuid: "02fcaedf69fe4ab7966328cc2c8abd5f" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 45571 } }
I20260926 06:14:59.408932 3061 catalog_manager.cc:5495] T 2d10c7194870412db75977141addcf21 P 399885d48c014147b59f3a4e18c080c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 399885d48c014147b59f3a4e18c080c6 (127.2.110.193). New cstate: current_term: 1 leader_uuid: "399885d48c014147b59f3a4e18c080c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d86f551cbc464cb3be3b10063e7934" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 39569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "399885d48c014147b59f3a4e18c080c6" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 38909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02fcaedf69fe4ab7966328cc2c8abd5f" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 45571 } health_report { overall_health: UNKNOWN } } }
W20260926 06:14:59.422039 3253 tablet.cc:2369] T 2d10c7194870412db75977141addcf21 P 399885d48c014147b59f3a4e18c080c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 06:14:59 dist-test-slave-hndd krb5kdc[3016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.110.193@KRBTEST.COM
W20260926 06:14:59.438181 3387 tablet.cc:2369] T 2d10c7194870412db75977141addcf21 P 67d86f551cbc464cb3be3b10063e7934: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:14:59.440508 3536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:14:59.432124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42458 (local address 127.2.110.193:38909)
0926 06:14:59.432211 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0926 06:14:59.432213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:14:59.432226 (+ 13us) server_negotiation.cc:407] Connection header received
0926 06:14:59.432302 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:14:59.432303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:14:59.432311 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:14:59.432357 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:14:59.433039 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:14:59.433654 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:14:59.434646 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:14:59.434880 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:14:59.434953 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:14:59.437257 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:14:59.437292 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:14:59.437293 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:14:59.437302 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:14:59.439257 (+ 1955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:14:59.439624 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:14:59.439627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:14:59.439627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:14:59.439703 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:14:59.439936 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:14:59.439936 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:14:59.439937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:14:59.440083 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:14:59.440225 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 06:14:59.440303 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 06:14:59.440375 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Sep 26 06:14:59 dist-test-slave-hndd krb5kdc[3016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403299, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260926 06:14:59.469089 3522 tablet.cc:2369] T 2d10c7194870412db75977141addcf21 P 02fcaedf69fe4ab7966328cc2c8abd5f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 06:14:59 dist-test-slave-hndd krb5kdc[3016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403299, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.110.193@KRBTEST.COM
I20260926 06:14:59.515787 3536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:14:59.506824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42472 (local address 127.2.110.193:38909)
0926 06:14:59.506891 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 06:14:59.506893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:14:59.507016 (+ 123us) server_negotiation.cc:407] Connection header received
0926 06:14:59.507085 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:14:59.507086 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:14:59.507098 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:14:59.507169 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:14:59.507914 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:14:59.508725 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:14:59.509570 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:14:59.509962 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:14:59.510038 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:14:59.513044 (+ 3006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:14:59.513045 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:14:59.513045 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:14:59.513053 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:14:59.514496 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:14:59.514844 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:14:59.514845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:14:59.514845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:14:59.514897 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:14:59.515218 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:14:59.515219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:14:59.515219 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:14:59.515335 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:14:59.515417 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 06:14:59.515613 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 06:14:59.515680 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
Sep 26 06:14:59 dist-test-slave-hndd krb5kdc[3016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 06:14:59 dist-test-slave-hndd krb5kdc[3016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.110.193@KRBTEST.COM
I20260926 06:14:59.613104 3536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:14:59.602900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42484 (local address 127.2.110.193:38909)
0926 06:14:59.603026 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 06:14:59.603028 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:14:59.603106 (+ 78us) server_negotiation.cc:407] Connection header received
0926 06:14:59.603155 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:14:59.603157 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:14:59.603169 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:14:59.603234 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:14:59.603846 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:14:59.604801 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:14:59.605971 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:14:59.606346 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:14:59.606434 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:14:59.609859 (+ 3425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:14:59.609860 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:14:59.609861 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:14:59.609871 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:14:59.611592 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:14:59.612176 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:14:59.612178 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:14:59.612178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:14:59.612287 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:14:59.612559 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:14:59.612560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:14:59.612560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:14:59.612693 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:14:59.612789 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 06:14:59.612958 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 06:14:59.613033 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
W20260926 06:14:59.613826 3166 tablet_service.cc:2908] Rejecting scan request for tablet 2d10c7194870412db75977141addcf21: Uninitialized: clean time has not yet been initialized
I20260926 06:14:59.614972 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 3123
I20260926 06:14:59.620813 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 3257
I20260926 06:14:59.625993 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 3392
I20260926 06:14:59.631148 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 3029
2026-09-26T06:14:59Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (5263 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:15:00 dist-test-slave-hndd krb5kdc[3565](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:15:00 dist-test-slave-hndd krb5kdc[3565](info): set up 2 sockets
Sep 26 06:15:00 dist-test-slave-hndd krb5kdc[3565](info): commencing operation
krb5kdc: starting...
W20260926 06:15:01.421074 2491 mini_kdc.cc:121] Time spent starting KDC: real 1.766s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Sep 26 06:15:01 dist-test-slave-hndd krb5kdc[3565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:15:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:15:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
I20260926 06:15:01.544574 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:34967
--webserver_interface=127.2.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:39041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.110.254:34967
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:15:01.630806 3578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:01.631088 3578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:01.631129 3578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:01.632418 3578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:15:01.632503 3578 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 06:15:01.632525 3578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:01.632550 3578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:01.632579 3578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:15:01.632609 3578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:15:01.634279 3578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:39041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.110.254:34967
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:34967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:01.634593 3578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:01.634850 3578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:01.640214 3583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:01.640246 3584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:01.640234 3586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:01.640692 3578 server_base.cc:1034] running on GCE node
I20260926 06:15:01.641005 3578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:01.641319 3578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:01.642560 3578 hybrid_clock.cc:648] HybridClock initialized: now 1790403301642538 us; error 40 us; skew 500 ppm
Sep 26 06:15:01 dist-test-slave-hndd krb5kdc[3565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403301, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:01.645608 3578 init.cc:377] Logged in from keytab as kudu/127.2.110.254@KRBTEST.COM (short username kudu)
I20260926 06:15:01.646093 3578 webserver.cc:460] Webserver started at http://127.2.110.254:33561/ using document root <none> and password file <none>
I20260926 06:15:01.646342 3578 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:01.646395 3578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:01.646521 3578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:01.647648 3578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/master-0/data/instance:
uuid: "1eff30d645784947bf39eafdb0868de2"
format_stamp: "Formatted at 2026-09-26 06:15:01 on dist-test-slave-hndd"
server_key: "fcfc23a192a754f402b53064e4334802"
server_key_iv: "8c8d6f48285ec98286705efe875c83d0"
server_key_version: "encryptionkey@0"
I20260926 06:15:01.648025 3578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/master-0/wal/instance:
uuid: "1eff30d645784947bf39eafdb0868de2"
format_stamp: "Formatted at 2026-09-26 06:15:01 on dist-test-slave-hndd"
server_key: "fcfc23a192a754f402b53064e4334802"
server_key_iv: "8c8d6f48285ec98286705efe875c83d0"
server_key_version: "encryptionkey@0"
I20260926 06:15:01.649730 3578 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260926 06:15:01.650872 3593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:01.651118 3578 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:01.651222 3578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "1eff30d645784947bf39eafdb0868de2"
format_stamp: "Formatted at 2026-09-26 06:15:01 on dist-test-slave-hndd"
server_key: "fcfc23a192a754f402b53064e4334802"
server_key_iv: "8c8d6f48285ec98286705efe875c83d0"
server_key_version: "encryptionkey@0"
I20260926 06:15:01.651304 3578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:01.662040 3578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:01.664428 3578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:01.664595 3578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:01.670607 3578 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.254:34967
I20260926 06:15:01.670648 3655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.254:34967 every 8 connection(s)
I20260926 06:15:01.671036 3578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:15:01.671747 3656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:01.674887 3656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1eff30d645784947bf39eafdb0868de2: Bootstrap starting.
I20260926 06:15:01.675670 3656 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1eff30d645784947bf39eafdb0868de2: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:01.676018 3656 log.cc:824] T 00000000000000000000000000000000 P 1eff30d645784947bf39eafdb0868de2: Log is configured to *not* fsync() on all Append() calls
I20260926 06:15:01.677059 3656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1eff30d645784947bf39eafdb0868de2: No bootstrap required, opened a new log
I20260926 06:15:01.679620 3656 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1eff30d645784947bf39eafdb0868de2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eff30d645784947bf39eafdb0868de2" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 34967 } }
I20260926 06:15:01.679801 3656 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1eff30d645784947bf39eafdb0868de2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:01.679827 3656 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1eff30d645784947bf39eafdb0868de2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eff30d645784947bf39eafdb0868de2, State: Initialized, Role: FOLLOWER
I20260926 06:15:01.680073 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 3578
I20260926 06:15:01.680070 3656 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1eff30d645784947bf39eafdb0868de2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eff30d645784947bf39eafdb0868de2" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 34967 } }
I20260926 06:15:01.680172 3656 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1eff30d645784947bf39eafdb0868de2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:15:01.680183 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/master-0/wal/instance
I20260926 06:15:01.680212 3656 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1eff30d645784947bf39eafdb0868de2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:15:01.680258 3656 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1eff30d645784947bf39eafdb0868de2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:01.680366 2491 external_mini_cluster.cc:1428] Setting key d6d6098bb88d7ede289f1a4ece196228
I20260926 06:15:01.681167 3656 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1eff30d645784947bf39eafdb0868de2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eff30d645784947bf39eafdb0868de2" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 34967 } }
I20260926 06:15:01.681301 3656 leader_election.cc:302] T 00000000000000000000000000000000 P 1eff30d645784947bf39eafdb0868de2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1eff30d645784947bf39eafdb0868de2; no voters:
I20260926 06:15:01.681526 3656 leader_election.cc:288] T 00000000000000000000000000000000 P 1eff30d645784947bf39eafdb0868de2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:15:01.681551 3661 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1eff30d645784947bf39eafdb0868de2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:15:01.681767 3661 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1eff30d645784947bf39eafdb0868de2 [term 1 LEADER]: Becoming Leader. State: Replica: 1eff30d645784947bf39eafdb0868de2, State: Running, Role: LEADER
I20260926 06:15:01.681895 3656 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1eff30d645784947bf39eafdb0868de2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:15:01.681931 3661 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1eff30d645784947bf39eafdb0868de2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eff30d645784947bf39eafdb0868de2" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 34967 } }
I20260926 06:15:01.682425 3662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1eff30d645784947bf39eafdb0868de2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1eff30d645784947bf39eafdb0868de2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eff30d645784947bf39eafdb0868de2" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 34967 } } }
I20260926 06:15:01.682528 3662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1eff30d645784947bf39eafdb0868de2 [sys.catalog]: This master's current role is: LEADER
I20260926 06:15:01.682798 3663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1eff30d645784947bf39eafdb0868de2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1eff30d645784947bf39eafdb0868de2. Latest consensus state: current_term: 1 leader_uuid: "1eff30d645784947bf39eafdb0868de2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eff30d645784947bf39eafdb0868de2" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 34967 } } }
I20260926 06:15:01.682926 3663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1eff30d645784947bf39eafdb0868de2 [sys.catalog]: This master's current role is: LEADER
I20260926 06:15:01.683074 3670 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:15:01.683600 3670 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:15:01.686245 3670 catalog_manager.cc:1347] Generated new cluster ID: a1c36ba066fe421ea0a5401c9aca787c
I20260926 06:15:01.686367 3670 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 06:15:01 dist-test-slave-hndd krb5kdc[3565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:15:01.693884 3660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:01.680951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51164 (local address 127.2.110.254:34967)
0926 06:15:01.681177 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:01.681191 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:01.681220 (+ 29us) server_negotiation.cc:407] Connection header received
0926 06:15:01.681377 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:01.681385 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:01.681449 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:01.681614 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:01.682351 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:01.683773 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:01.684912 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:01.685519 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:01.685647 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:01.689020 (+ 3373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:01.689025 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:01.689026 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:01.689038 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:01.691883 (+ 2845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:01.692390 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:01.692393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:01.692394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:01.692502 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:01.692746 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:01.692746 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:01.692747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:01.693070 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:01.693231 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:01.693388 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 06:15:01.693646 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":104,"threads_started":1}
I20260926 06:15:01.722142 3670 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:15:01.722781 3670 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:15:01.726063 3670 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1eff30d645784947bf39eafdb0868de2: Generated new TSK 0
I20260926 06:15:01.726280 3670 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab.
Entry for principal kudu/127.2.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab.
I20260926 06:15:01.760526 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.193:0
--local_ip_for_outbound_sockets=127.2.110.193
--webserver_interface=127.2.110.193
--webserver_port=0
--tserver_master_addrs=127.2.110.254:34967
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab
--principal=kudu/127.2.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:39041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:15:01.848927 3682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:01.849172 3682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:01.849200 3682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:01.851524 3682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:01.851642 3682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:01.851675 3682 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 06:15:01.851701 3682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.193
I20260926 06:15:01.854513 3682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:39041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab
--principal=kudu/127.2.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.110.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.110.254:34967
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.193
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:01.854763 3682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:01.854961 3682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:01.860407 3687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:01.860400 3688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:01.860435 3682 server_base.cc:1034] running on GCE node
W20260926 06:15:01.860401 3690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:01.860846 3682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:01.861071 3682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:01.862247 3682 hybrid_clock.cc:648] HybridClock initialized: now 1790403301862238 us; error 49 us; skew 500 ppm
Sep 26 06:15:01 dist-test-slave-hndd krb5kdc[3565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403301, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:01.865113 3682 init.cc:377] Logged in from keytab as kudu/127.2.110.193@KRBTEST.COM (short username kudu)
I20260926 06:15:01.865792 3682 webserver.cc:460] Webserver started at http://127.2.110.193:43857/ using document root <none> and password file <none>
I20260926 06:15:01.866058 3682 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:01.866144 3682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:01.866320 3682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:01.867568 3682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/ts-0/data/instance:
uuid: "f9895ecbf03947dc8d50afde1f9f9a1e"
format_stamp: "Formatted at 2026-09-26 06:15:01 on dist-test-slave-hndd"
server_key: "aa957185275d8885a9ebf4f1c7aa8f0c"
server_key_iv: "05ba445be8a573a912ab35a33202d95e"
server_key_version: "encryptionkey@0"
I20260926 06:15:01.868002 3682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/ts-0/wal/instance:
uuid: "f9895ecbf03947dc8d50afde1f9f9a1e"
format_stamp: "Formatted at 2026-09-26 06:15:01 on dist-test-slave-hndd"
server_key: "aa957185275d8885a9ebf4f1c7aa8f0c"
server_key_iv: "05ba445be8a573a912ab35a33202d95e"
server_key_version: "encryptionkey@0"
I20260926 06:15:01.869614 3682 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:15:01.870668 3697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:01.870983 3682 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:01.871222 3682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/ts-0/wal
uuid: "f9895ecbf03947dc8d50afde1f9f9a1e"
format_stamp: "Formatted at 2026-09-26 06:15:01 on dist-test-slave-hndd"
server_key: "aa957185275d8885a9ebf4f1c7aa8f0c"
server_key_iv: "05ba445be8a573a912ab35a33202d95e"
server_key_version: "encryptionkey@0"
I20260926 06:15:01.871469 3682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:01.887763 3682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:01.890487 3682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:01.890681 3682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:01.891336 3682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:15:01.891392 3682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:01.891424 3682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:15:01.891443 3682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 06:15:01 dist-test-slave-hndd krb5kdc[3565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403301, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.193@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:15:01.900681 3682 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.193:36493
I20260926 06:15:01.900740 3812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.193:36493 every 8 connection(s)
I20260926 06:15:01.901098 3682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
I20260926 06:15:01.903594 3660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:01.891775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:41773 (local address 127.2.110.254:34967)
0926 06:15:01.891849 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:01.891853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:01.892180 (+ 327us) server_negotiation.cc:407] Connection header received
0926 06:15:01.892702 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:01.892704 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:01.892726 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:01.892853 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:01.893658 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:01.894586 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:01.896438 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:01.896898 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:01.897041 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:01.900234 (+ 3193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:01.900237 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:01.900237 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:01.900247 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:01.901927 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:01.902553 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:01.902554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:01.902554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:01.902614 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:01.902947 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:01.902947 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:01.902948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:01.903068 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:01.903122 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:01.903448 (+ 326us) server_negotiation.cc:299] Negotiation successful
0926 06:15:01.903527 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 06:15:01.903885 3706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:01.891787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34967 (local address 127.2.110.193:41773)
0926 06:15:01.891974 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:01.892019 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:01.892572 (+ 553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:01.892944 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:01.892947 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:01.893233 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:01.893497 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:01.893503 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:01.894752 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:01.894753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:01.896236 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:01.896242 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:01.897074 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:01.897076 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:01.897269 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:01.898025 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:01.898034 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:01.900113 (+ 2079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:01.902073 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:01.902091 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:01.902094 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:01.902444 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:01.902727 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:01.902728 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:01.902728 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:01.902873 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:01.903172 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:01.903178 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:01.903317 (+ 139us) client_negotiation.cc:769] Sending connection context
0926 06:15:01.903418 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 06:15:01.903572 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":99,"threads_started":1}
I20260926 06:15:01.904387 3813 heartbeater.cc:347] Connected to a master server at 127.2.110.254:34967
I20260926 06:15:01.904467 3813 heartbeater.cc:464] Registering TS with master...
I20260926 06:15:01.904662 3813 heartbeater.cc:511] Master 127.2.110.254:34967 requested a full tablet report, sending...
I20260926 06:15:01.905294 3609 ts_manager.cc:194] Registered new tserver with Master: f9895ecbf03947dc8d50afde1f9f9a1e (127.2.110.193:36493)
I20260926 06:15:01.906057 3609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.193@KRBTEST.COM'} at 127.2.110.193:41773
I20260926 06:15:01.906433 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 3682
I20260926 06:15:01.906538 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/ts-0/wal/instance
I20260926 06:15:01.906744 2491 external_mini_cluster.cc:1428] Setting key 80bf5baf0d77a2af83c1dedbed80a526
I20260926 06:15:01.914216 3660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:01.905922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:53841 (local address 127.2.110.254:34967)
0926 06:15:01.906008 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:01.906011 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:01.906288 (+ 277us) server_negotiation.cc:407] Connection header received
0926 06:15:01.906333 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:01.906335 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:01.906349 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:01.906440 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:01.907180 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:01.907902 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:01.909063 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:01.909604 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:01.909693 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:01.910780 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:01.910781 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:01.910782 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:01.910790 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:01.912775 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:01.913280 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:01.913281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:01.913281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:01.913379 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:01.913602 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:01.913602 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:01.913603 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:01.913718 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:01.913771 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:01.913987 (+ 216us) server_negotiation.cc:299] Negotiation successful
0926 06:15:01.914056 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 06:15:01.914270 3819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:01.905848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34967 (local address 127.2.110.193:53841)
0926 06:15:01.906061 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:01.906076 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:01.906275 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:01.906457 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:01.906458 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:01.906756 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:01.906893 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:01.906896 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:01.908010 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:01.908011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:01.908920 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:01.908923 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:01.909699 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:01.909700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:01.909766 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:01.910359 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:01.910364 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:01.910694 (+ 330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:01.912883 (+ 2189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:01.912885 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:01.912885 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:01.913191 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:01.913443 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:01.913444 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:01.913445 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:01.913529 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:01.913789 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:01.913791 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:01.913854 (+ 63us) client_negotiation.cc:769] Sending connection context
0926 06:15:01.913970 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 06:15:01.914112 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":140,"threads_started":1}
WARNING: no policy specified for kudu/127.2.110.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab.
Entry for principal kudu/127.2.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab.
I20260926 06:15:01.936514 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.194:0
--local_ip_for_outbound_sockets=127.2.110.194
--webserver_interface=127.2.110.194
--webserver_port=0
--tserver_master_addrs=127.2.110.254:34967
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab
--principal=kudu/127.2.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:39041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:15:02.028508 3822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:02.028832 3822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:02.028867 3822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:02.030999 3822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:02.031105 3822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:02.031141 3822 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 06:15:02.031165 3822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.194
I20260926 06:15:02.033291 3822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:39041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab
--principal=kudu/127.2.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.110.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.110.254:34967
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.194
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:02.033614 3822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:02.033823 3822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:02.039471 3828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:02.039482 3830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:02.039498 3822 server_base.cc:1034] running on GCE node
W20260926 06:15:02.039479 3827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:02.039912 3822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:02.040285 3822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:02.041483 3822 hybrid_clock.cc:648] HybridClock initialized: now 1790403302041469 us; error 36 us; skew 500 ppm
Sep 26 06:15:02 dist-test-slave-hndd krb5kdc[3565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403302, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:02.044210 3822 init.cc:377] Logged in from keytab as kudu/127.2.110.194@KRBTEST.COM (short username kudu)
I20260926 06:15:02.044723 3822 webserver.cc:460] Webserver started at http://127.2.110.194:46407/ using document root <none> and password file <none>
I20260926 06:15:02.044929 3822 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:02.044976 3822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:02.045070 3822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:02.046240 3822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/ts-1/data/instance:
uuid: "e758657151684f7bac40258b0e41a0c7"
format_stamp: "Formatted at 2026-09-26 06:15:02 on dist-test-slave-hndd"
server_key: "c49f3229811ffd58cb6be42a69e40632"
server_key_iv: "3eeeb47b11a29abe180f794b25059dc0"
server_key_version: "encryptionkey@0"
I20260926 06:15:02.046571 3822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/ts-1/wal/instance:
uuid: "e758657151684f7bac40258b0e41a0c7"
format_stamp: "Formatted at 2026-09-26 06:15:02 on dist-test-slave-hndd"
server_key: "c49f3229811ffd58cb6be42a69e40632"
server_key_iv: "3eeeb47b11a29abe180f794b25059dc0"
server_key_version: "encryptionkey@0"
I20260926 06:15:02.047945 3822 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:15:02.048689 3837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:02.048915 3822 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:02.049026 3822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/ts-1/wal
uuid: "e758657151684f7bac40258b0e41a0c7"
format_stamp: "Formatted at 2026-09-26 06:15:02 on dist-test-slave-hndd"
server_key: "c49f3229811ffd58cb6be42a69e40632"
server_key_iv: "3eeeb47b11a29abe180f794b25059dc0"
server_key_version: "encryptionkey@0"
I20260926 06:15:02.049135 3822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:02.055822 3822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:02.057869 3822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:02.057986 3822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:02.058475 3822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:15:02.058537 3822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:02.058584 3822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:15:02.058635 3822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:02.065374 3822 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.194:36241
I20260926 06:15:02.065492 3952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.194:36241 every 8 connection(s)
I20260926 06:15:02.065829 3822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
Sep 26 06:15:02 dist-test-slave-hndd krb5kdc[3565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403302, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.194@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:15:02.069849 3660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:02.059115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:59265 (local address 127.2.110.254:34967)
0926 06:15:02.059158 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:02.059160 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:02.059422 (+ 262us) server_negotiation.cc:407] Connection header received
0926 06:15:02.059996 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:02.059997 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:02.060008 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:02.060054 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:02.060969 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.061842 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.063523 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.063855 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.063945 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:02.066819 (+ 2874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:02.066821 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:02.066821 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:02.066828 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:02.068055 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:02.068724 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:02.068724 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:02.068725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:02.068781 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:02.069199 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:02.069200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:02.069200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:02.069316 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:02.069448 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:02.069689 (+ 241us) server_negotiation.cc:299] Negotiation successful
0926 06:15:02.069759 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 06:15:02.070046 3846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:02.059134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34967 (local address 127.2.110.194:59265)
0926 06:15:02.059288 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:02.059332 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:02.059894 (+ 562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:02.060122 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:02.060126 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:02.060530 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:02.060848 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:02.060855 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.061943 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.061944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:02.063409 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:02.063412 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.064028 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.064029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:02.064201 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:02.064895 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:02.064903 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:02.066728 (+ 1825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:02.068213 (+ 1485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:02.068227 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:02.068231 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:02.068615 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:02.068852 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:02.068853 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:02.068853 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:02.069109 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:02.069439 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:02.069444 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:02.069572 (+ 128us) client_negotiation.cc:769] Sending connection context
0926 06:15:02.069664 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 06:15:02.069808 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":83,"threads_started":1}
I20260926 06:15:02.070631 3953 heartbeater.cc:347] Connected to a master server at 127.2.110.254:34967
I20260926 06:15:02.070724 3953 heartbeater.cc:464] Registering TS with master...
I20260926 06:15:02.070945 3953 heartbeater.cc:511] Master 127.2.110.254:34967 requested a full tablet report, sending...
I20260926 06:15:02.071365 3609 ts_manager.cc:194] Registered new tserver with Master: e758657151684f7bac40258b0e41a0c7 (127.2.110.194:36241)
I20260926 06:15:02.071844 3609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.194@KRBTEST.COM'} at 127.2.110.194:59265
I20260926 06:15:02.072239 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 3822
I20260926 06:15:02.072353 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/ts-1/wal/instance
I20260926 06:15:02.072556 2491 external_mini_cluster.cc:1428] Setting key eeb51803ab35d772e141ce0043ce2c18
I20260926 06:15:02.079141 3660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:02.071736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:52129 (local address 127.2.110.254:34967)
0926 06:15:02.071790 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:02.071791 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:02.071910 (+ 119us) server_negotiation.cc:407] Connection header received
0926 06:15:02.072010 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:02.072011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:02.072023 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:02.072090 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:02.072490 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.073137 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.074203 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.074414 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.074488 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:02.075984 (+ 1496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:02.075987 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:02.075987 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:02.075997 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:02.077870 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:02.078285 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:02.078286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:02.078286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:02.078348 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:02.078549 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:02.078550 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:02.078550 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:02.078716 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:02.078797 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:02.078948 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 06:15:02.079035 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 06:15:02.079155 3959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:02.071685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34967 (local address 127.2.110.194:52129)
0926 06:15:02.071848 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:02.071857 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:02.071924 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:02.072101 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:02.072102 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:02.072331 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:02.072404 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:02.072407 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.073208 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.073209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:02.074100 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:02.074103 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.074520 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.074521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:02.074604 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:02.075230 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:02.075236 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:02.075809 (+ 573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:02.077988 (+ 2179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:02.077990 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:02.077991 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:02.078209 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:02.078423 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:02.078423 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:02.078424 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:02.078488 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:02.078787 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:02.078788 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:02.078841 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 06:15:02.078924 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 06:15:02.079038 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":70,"threads_started":1}
WARNING: no policy specified for kudu/127.2.110.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab.
Entry for principal kudu/127.2.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab.
I20260926 06:15:02.100446 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.195:0
--local_ip_for_outbound_sockets=127.2.110.195
--webserver_interface=127.2.110.195
--webserver_port=0
--tserver_master_addrs=127.2.110.254:34967
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab
--principal=kudu/127.2.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:39041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:15:02.184952 3962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:02.185175 3962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:02.185201 3962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:02.186527 3962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:02.186580 3962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:02.186604 3962 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 06:15:02.186622 3962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.195
I20260926 06:15:02.188100 3962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:39041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab
--principal=kudu/127.2.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.110.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.110.254:34967
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.195
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:02.188359 3962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:02.188589 3962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:02.193670 3968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:02.193645 3970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:02.193674 3967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:02.193970 3962 server_base.cc:1034] running on GCE node
I20260926 06:15:02.194167 3962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:02.194423 3962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:02.195600 3962 hybrid_clock.cc:648] HybridClock initialized: now 1790403302195575 us; error 40 us; skew 500 ppm
Sep 26 06:15:02 dist-test-slave-hndd krb5kdc[3565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403302, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:02.198040 3962 init.cc:377] Logged in from keytab as kudu/127.2.110.195@KRBTEST.COM (short username kudu)
I20260926 06:15:02.198496 3962 webserver.cc:460] Webserver started at http://127.2.110.195:37481/ using document root <none> and password file <none>
I20260926 06:15:02.198724 3962 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:02.198796 3962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:02.198946 3962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:02.200022 3962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/ts-2/data/instance:
uuid: "a2a43bf937df4caf8fa833e7f9069167"
format_stamp: "Formatted at 2026-09-26 06:15:02 on dist-test-slave-hndd"
server_key: "5cc5532e5cf61bb2e41a1278b82d1aa3"
server_key_iv: "c6f241a6a17cf46802d77732d05dd50a"
server_key_version: "encryptionkey@0"
I20260926 06:15:02.200388 3962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/ts-2/wal/instance:
uuid: "a2a43bf937df4caf8fa833e7f9069167"
format_stamp: "Formatted at 2026-09-26 06:15:02 on dist-test-slave-hndd"
server_key: "5cc5532e5cf61bb2e41a1278b82d1aa3"
server_key_iv: "c6f241a6a17cf46802d77732d05dd50a"
server_key_version: "encryptionkey@0"
I20260926 06:15:02.201793 3962 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 06:15:02.202585 3977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:02.202796 3962 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:02.202891 3962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/ts-2/wal
uuid: "a2a43bf937df4caf8fa833e7f9069167"
format_stamp: "Formatted at 2026-09-26 06:15:02 on dist-test-slave-hndd"
server_key: "5cc5532e5cf61bb2e41a1278b82d1aa3"
server_key_iv: "c6f241a6a17cf46802d77732d05dd50a"
server_key_version: "encryptionkey@0"
I20260926 06:15:02.202970 3962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:02.220888 3962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:02.223410 3962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:02.223603 3962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:02.224198 3962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:15:02.224256 3962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:02.224294 3962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:15:02.224331 3962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 06:15:02 dist-test-slave-hndd krb5kdc[3565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403302, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.195@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:15:02.231824 3962 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.195:40247
I20260926 06:15:02.232184 3962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
I20260926 06:15:02.232177 4092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.195:40247 every 8 connection(s)
I20260926 06:15:02.234148 3660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:02.224659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:42815 (local address 127.2.110.254:34967)
0926 06:15:02.224731 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:02.224734 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:02.224965 (+ 231us) server_negotiation.cc:407] Connection header received
0926 06:15:02.225508 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:02.225510 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:02.225524 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:02.225608 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:02.226469 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.227074 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.228109 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.228313 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.228396 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:02.231032 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:02.231034 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:02.231034 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:02.231042 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:02.232711 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:02.233153 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:02.233154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:02.233155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:02.233212 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:02.233493 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:02.233494 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:02.233494 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:02.233622 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:02.233695 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:02.233975 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 06:15:02.234058 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 06:15:02.234356 3986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:02.224645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34967 (local address 127.2.110.195:42815)
0926 06:15:02.224830 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:02.224870 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:02.225400 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:02.225710 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:02.225718 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:02.226159 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:02.226376 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:02.226381 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.227169 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.227170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:02.228015 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:02.228017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.228403 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.228404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:02.228570 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:02.229140 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:02.229153 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:02.230924 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:02.232809 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:02.232815 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:02.232817 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:02.233077 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:02.233284 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:02.233284 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:02.233285 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:02.233427 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:02.233685 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:02.233692 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:02.233839 (+ 147us) client_negotiation.cc:769] Sending connection context
0926 06:15:02.233938 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 06:15:02.234106 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":108,"threads_started":1}
I20260926 06:15:02.234802 4093 heartbeater.cc:347] Connected to a master server at 127.2.110.254:34967
I20260926 06:15:02.234884 4093 heartbeater.cc:464] Registering TS with master...
I20260926 06:15:02.235092 4093 heartbeater.cc:511] Master 127.2.110.254:34967 requested a full tablet report, sending...
I20260926 06:15:02.235610 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 3962
I20260926 06:15:02.235682 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/ts-2/wal/instance
I20260926 06:15:02.235862 2491 external_mini_cluster.cc:1428] Setting key 76ef790476dc3198ce30385292073089
I20260926 06:15:02.236184 3610 ts_manager.cc:194] Registered new tserver with Master: a2a43bf937df4caf8fa833e7f9069167 (127.2.110.195:40247)
I20260926 06:15:02.236579 3610 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.195@KRBTEST.COM'} at 127.2.110.195:42815
I20260926 06:15:02.237634 2491 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 06:15:02.242169 4099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:02.235872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34967 (local address 127.2.110.195:39797)
0926 06:15:02.236004 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:02.236014 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:02.236061 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:02.236249 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:02.236250 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:02.236442 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:02.236501 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:02.236503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.237435 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.237436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:02.238084 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:02.238087 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.238552 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.238553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:02.238608 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:02.239082 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:02.239087 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:02.239394 (+ 307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:02.241147 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:02.241148 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:02.241149 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:02.241326 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:02.241564 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:02.241564 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:02.241564 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:02.241621 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:02.241915 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:02.241916 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:02.241957 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 06:15:02.242010 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 06:15:02.242096 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":81,"threads_started":1}
I20260926 06:15:02.242223 3660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:02.235945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:39797 (local address 127.2.110.254:34967)
0926 06:15:02.236135 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:02.236137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:02.236143 (+ 6us) server_negotiation.cc:407] Connection header received
0926 06:15:02.236159 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:02.236159 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:02.236168 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:02.236229 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:02.236578 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.237280 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.238162 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.238474 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.238551 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:02.239465 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:02.239466 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:02.239467 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:02.239475 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:02.241058 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:02.241434 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:02.241435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:02.241436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:02.241497 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:02.241683 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:02.241684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:02.241684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:02.241847 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:02.241928 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:02.242035 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 06:15:02.242119 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170}
Sep 26 06:15:02 dist-test-slave-hndd krb5kdc[3565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403302, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 06:15:02 dist-test-slave-hndd krb5kdc[3565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403302, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:15:02.262306 3660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:02.255320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51166 (local address 127.2.110.254:34967)
0926 06:15:02.255365 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:02.255368 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:02.255375 (+ 7us) server_negotiation.cc:407] Connection header received
0926 06:15:02.255400 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:02.255401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:02.255410 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:02.255470 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:02.256015 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.256498 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.257235 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.257493 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.257550 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:02.259853 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:02.259854 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:02.259854 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:02.259861 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:02.261173 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:02.261593 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:02.261594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:02.261595 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:02.261649 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:02.261806 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:02.261807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:02.261807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:02.261908 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:02.261982 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:02.262172 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 06:15:02.262237 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 06:15:02.264957 3610 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51166:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:15:02.267697 3610 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:15:02.275264 4111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:02.270692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.194:36241 (local address 127.0.0.1:60726)
0926 06:15:02.271088 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:02.271099 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:02.271179 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:02.271580 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:02.271584 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:02.271587 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:15:02.271684 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:02.271687 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.273106 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.273106 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:02.274216 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:02.274218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.274943 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.274944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:02.275024 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:02.275038 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 06:15:02.275108 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 06:15:02.275164 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":154,"threads_started":1}
I20260926 06:15:02.275547 4113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:02.270880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54332 (local address 127.2.110.195:40247)
0926 06:15:02.271552 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:02.271557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:02.271570 (+ 13us) server_negotiation.cc:407] Connection header received
0926 06:15:02.271611 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:02.271613 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:02.271629 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:02.271721 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:15:02.271944 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.272664 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.274202 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.275025 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.275090 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:02.275146 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:02.275280 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 06:15:02.275324 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":89,"threads_started":1}
I20260926 06:15:02.275553 4115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:02.271199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56140 (local address 127.2.110.193:36493)
0926 06:15:02.271453 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:02.271456 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:02.271466 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:15:02.271502 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:02.271504 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:02.271518 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:02.271764 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:15:02.271864 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.273012 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.274225 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.275009 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.275320 (+ 311us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:02.275378 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:02.275405 (+ 27us) server_negotiation.cc:299] Negotiation successful
0926 06:15:02.275454 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":115,"threads_started":1}
I20260926 06:15:02.275679 4114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:02.270958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60726 (local address 127.2.110.194:36241)
0926 06:15:02.271379 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:02.271385 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:02.271396 (+ 11us) server_negotiation.cc:407] Connection header received
0926 06:15:02.271439 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:02.271441 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:02.271459 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:02.271771 (+ 312us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:15:02.271836 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.273011 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.274291 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.274873 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.274944 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:02.275006 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:02.275503 (+ 497us) server_negotiation.cc:299] Negotiation successful
0926 06:15:02.275590 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":124,"threads_started":1}
I20260926 06:15:02.275580 4110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:02.270692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.193:36493 (local address 127.0.0.1:56140)
0926 06:15:02.271058 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:02.271101 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:02.271179 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:02.271638 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:02.271638 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:02.271640 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:15:02.271683 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:02.271686 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.273130 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.273131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:02.274114 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:02.274117 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.275107 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.275109 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:02.275187 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:02.275193 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 06:15:02.275225 (+ 32us) client_negotiation.cc:240] Negotiation successful
0926 06:15:02.275273 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":250,"threads_started":1}
I20260926 06:15:02.276101 4112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:02.270659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.195:40247 (local address 127.0.0.1:54332)
0926 06:15:02.271038 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:02.271097 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:02.271182 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:02.271803 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:02.271804 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:02.271806 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:15:02.271855 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:02.271857 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.272765 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.272766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:02.274106 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:02.274109 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.275092 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.275093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:02.275148 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:02.275150 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 06:15:02.275216 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 06:15:02.275250 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":61,"threads_started":1}
I20260926 06:15:02.276468 3743 tablet_service.cc:1467] Processing CreateTablet for tablet 32f72973923a4d32b640b85aa5bbe0b4 (DEFAULT_TABLE table=test-table [id=44d36946593e4c0e823d222ca40b8011]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:15:02.276427 3887 tablet_service.cc:1467] Processing CreateTablet for tablet 32f72973923a4d32b640b85aa5bbe0b4 (DEFAULT_TABLE table=test-table [id=44d36946593e4c0e823d222ca40b8011]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:15:02.276662 4027 tablet_service.cc:1467] Processing CreateTablet for tablet 32f72973923a4d32b640b85aa5bbe0b4 (DEFAULT_TABLE table=test-table [id=44d36946593e4c0e823d222ca40b8011]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:15:02.276810 3887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32f72973923a4d32b640b85aa5bbe0b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:02.276810 3743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32f72973923a4d32b640b85aa5bbe0b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:02.276913 4027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32f72973923a4d32b640b85aa5bbe0b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:02.279743 4116 tablet_bootstrap.cc:492] T 32f72973923a4d32b640b85aa5bbe0b4 P a2a43bf937df4caf8fa833e7f9069167: Bootstrap starting.
I20260926 06:15:02.280503 4118 tablet_bootstrap.cc:492] T 32f72973923a4d32b640b85aa5bbe0b4 P e758657151684f7bac40258b0e41a0c7: Bootstrap starting.
I20260926 06:15:02.280916 4116 tablet_bootstrap.cc:654] T 32f72973923a4d32b640b85aa5bbe0b4 P a2a43bf937df4caf8fa833e7f9069167: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:02.281164 4118 tablet_bootstrap.cc:654] T 32f72973923a4d32b640b85aa5bbe0b4 P e758657151684f7bac40258b0e41a0c7: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:02.281450 4116 log.cc:824] T 32f72973923a4d32b640b85aa5bbe0b4 P a2a43bf937df4caf8fa833e7f9069167: Log is configured to *not* fsync() on all Append() calls
I20260926 06:15:02.281494 4117 tablet_bootstrap.cc:492] T 32f72973923a4d32b640b85aa5bbe0b4 P f9895ecbf03947dc8d50afde1f9f9a1e: Bootstrap starting.
I20260926 06:15:02.281545 4118 log.cc:824] T 32f72973923a4d32b640b85aa5bbe0b4 P e758657151684f7bac40258b0e41a0c7: Log is configured to *not* fsync() on all Append() calls
I20260926 06:15:02.282137 4117 tablet_bootstrap.cc:654] T 32f72973923a4d32b640b85aa5bbe0b4 P f9895ecbf03947dc8d50afde1f9f9a1e: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:02.282383 4118 tablet_bootstrap.cc:492] T 32f72973923a4d32b640b85aa5bbe0b4 P e758657151684f7bac40258b0e41a0c7: No bootstrap required, opened a new log
I20260926 06:15:02.282464 4118 ts_tablet_manager.cc:1397] T 32f72973923a4d32b640b85aa5bbe0b4 P e758657151684f7bac40258b0e41a0c7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 06:15:02.282487 4117 log.cc:824] T 32f72973923a4d32b640b85aa5bbe0b4 P f9895ecbf03947dc8d50afde1f9f9a1e: Log is configured to *not* fsync() on all Append() calls
I20260926 06:15:02.282645 4116 tablet_bootstrap.cc:492] T 32f72973923a4d32b640b85aa5bbe0b4 P a2a43bf937df4caf8fa833e7f9069167: No bootstrap required, opened a new log
I20260926 06:15:02.282776 4116 ts_tablet_manager.cc:1397] T 32f72973923a4d32b640b85aa5bbe0b4 P a2a43bf937df4caf8fa833e7f9069167: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.002s
I20260926 06:15:02.283339 4117 tablet_bootstrap.cc:492] T 32f72973923a4d32b640b85aa5bbe0b4 P f9895ecbf03947dc8d50afde1f9f9a1e: No bootstrap required, opened a new log
I20260926 06:15:02.283442 4117 ts_tablet_manager.cc:1397] T 32f72973923a4d32b640b85aa5bbe0b4 P f9895ecbf03947dc8d50afde1f9f9a1e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:15:02.284161 4118 raft_consensus.cc:348] T 32f72973923a4d32b640b85aa5bbe0b4 P e758657151684f7bac40258b0e41a0c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a43bf937df4caf8fa833e7f9069167" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 40247 } } peers { permanent_uuid: "f9895ecbf03947dc8d50afde1f9f9a1e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 36493 } } peers { permanent_uuid: "e758657151684f7bac40258b0e41a0c7" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36241 } }
I20260926 06:15:02.284301 4118 raft_consensus.cc:374] T 32f72973923a4d32b640b85aa5bbe0b4 P e758657151684f7bac40258b0e41a0c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:02.284346 4118 raft_consensus.cc:729] T 32f72973923a4d32b640b85aa5bbe0b4 P e758657151684f7bac40258b0e41a0c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e758657151684f7bac40258b0e41a0c7, State: Initialized, Role: FOLLOWER
I20260926 06:15:02.284473 4118 consensus_queue.cc:260] T 32f72973923a4d32b640b85aa5bbe0b4 P e758657151684f7bac40258b0e41a0c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a43bf937df4caf8fa833e7f9069167" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 40247 } } peers { permanent_uuid: "f9895ecbf03947dc8d50afde1f9f9a1e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 36493 } } peers { permanent_uuid: "e758657151684f7bac40258b0e41a0c7" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36241 } }
I20260926 06:15:02.284755 4118 ts_tablet_manager.cc:1428] T 32f72973923a4d32b640b85aa5bbe0b4 P e758657151684f7bac40258b0e41a0c7: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260926 06:15:02.284777 3953 heartbeater.cc:503] Master 127.2.110.254:34967 was elected leader, sending a full tablet report...
I20260926 06:15:02.284842 4116 raft_consensus.cc:348] T 32f72973923a4d32b640b85aa5bbe0b4 P a2a43bf937df4caf8fa833e7f9069167 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a43bf937df4caf8fa833e7f9069167" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 40247 } } peers { permanent_uuid: "f9895ecbf03947dc8d50afde1f9f9a1e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 36493 } } peers { permanent_uuid: "e758657151684f7bac40258b0e41a0c7" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36241 } }
I20260926 06:15:02.285010 4116 raft_consensus.cc:374] T 32f72973923a4d32b640b85aa5bbe0b4 P a2a43bf937df4caf8fa833e7f9069167 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:02.285048 4116 raft_consensus.cc:729] T 32f72973923a4d32b640b85aa5bbe0b4 P a2a43bf937df4caf8fa833e7f9069167 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a43bf937df4caf8fa833e7f9069167, State: Initialized, Role: FOLLOWER
I20260926 06:15:02.285164 4116 consensus_queue.cc:260] T 32f72973923a4d32b640b85aa5bbe0b4 P a2a43bf937df4caf8fa833e7f9069167 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a43bf937df4caf8fa833e7f9069167" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 40247 } } peers { permanent_uuid: "f9895ecbf03947dc8d50afde1f9f9a1e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 36493 } } peers { permanent_uuid: "e758657151684f7bac40258b0e41a0c7" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36241 } }
I20260926 06:15:02.285440 4116 ts_tablet_manager.cc:1428] T 32f72973923a4d32b640b85aa5bbe0b4 P a2a43bf937df4caf8fa833e7f9069167: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 06:15:02.285473 4093 heartbeater.cc:503] Master 127.2.110.254:34967 was elected leader, sending a full tablet report...
I20260926 06:15:02.286296 4123 raft_consensus.cc:482] T 32f72973923a4d32b640b85aa5bbe0b4 P a2a43bf937df4caf8fa833e7f9069167 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:15:02.286196 4117 raft_consensus.cc:348] T 32f72973923a4d32b640b85aa5bbe0b4 P f9895ecbf03947dc8d50afde1f9f9a1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a43bf937df4caf8fa833e7f9069167" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 40247 } } peers { permanent_uuid: "f9895ecbf03947dc8d50afde1f9f9a1e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 36493 } } peers { permanent_uuid: "e758657151684f7bac40258b0e41a0c7" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36241 } }
I20260926 06:15:02.286352 4117 raft_consensus.cc:374] T 32f72973923a4d32b640b85aa5bbe0b4 P f9895ecbf03947dc8d50afde1f9f9a1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:02.286370 4123 raft_consensus.cc:504] T 32f72973923a4d32b640b85aa5bbe0b4 P a2a43bf937df4caf8fa833e7f9069167 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a43bf937df4caf8fa833e7f9069167" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 40247 } } peers { permanent_uuid: "f9895ecbf03947dc8d50afde1f9f9a1e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 36493 } } peers { permanent_uuid: "e758657151684f7bac40258b0e41a0c7" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36241 } }
I20260926 06:15:02.286383 4117 raft_consensus.cc:729] T 32f72973923a4d32b640b85aa5bbe0b4 P f9895ecbf03947dc8d50afde1f9f9a1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9895ecbf03947dc8d50afde1f9f9a1e, State: Initialized, Role: FOLLOWER
I20260926 06:15:02.286527 4117 consensus_queue.cc:260] T 32f72973923a4d32b640b85aa5bbe0b4 P f9895ecbf03947dc8d50afde1f9f9a1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a43bf937df4caf8fa833e7f9069167" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 40247 } } peers { permanent_uuid: "f9895ecbf03947dc8d50afde1f9f9a1e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 36493 } } peers { permanent_uuid: "e758657151684f7bac40258b0e41a0c7" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36241 } }
I20260926 06:15:02.286799 4117 ts_tablet_manager.cc:1428] T 32f72973923a4d32b640b85aa5bbe0b4 P f9895ecbf03947dc8d50afde1f9f9a1e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 06:15:02.287381 3813 heartbeater.cc:503] Master 127.2.110.254:34967 was elected leader, sending a full tablet report...
I20260926 06:15:02.287616 4123 leader_election.cc:288] T 32f72973923a4d32b640b85aa5bbe0b4 P a2a43bf937df4caf8fa833e7f9069167 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9895ecbf03947dc8d50afde1f9f9a1e (127.2.110.193:36493), e758657151684f7bac40258b0e41a0c7 (127.2.110.194:36241)
I20260926 06:15:02.289855 4115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:02.286968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:45167 (local address 127.2.110.193:36493)
0926 06:15:02.287060 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:02.287062 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:02.287154 (+ 92us) server_negotiation.cc:407] Connection header received
0926 06:15:02.287185 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:02.287186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:02.287195 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:02.287270 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:15:02.287443 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.288064 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.289086 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.289509 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.289568 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:02.289575 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:02.289745 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 06:15:02.289791 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 06:15:02.289934 4114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:02.287017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:38783 (local address 127.2.110.194:36241)
0926 06:15:02.287092 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:02.287094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:02.287103 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:15:02.287127 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:02.287128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:02.287137 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:02.287182 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:15:02.287349 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.288005 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.288972 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.289567 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.289647 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:02.289666 (+ 19us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:02.289821 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 06:15:02.289861 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 06:15:02.289955 3986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:02.286882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.194:36241 (local address 127.2.110.195:38783)
0926 06:15:02.287004 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:02.287016 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:02.287056 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:02.287186 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:02.287187 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:02.287188 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:15:02.287272 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:02.287274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.288071 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.288072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:02.288889 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:02.288892 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.289650 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.289651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:02.289749 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:02.289752 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:15:02.289808 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 06:15:02.289847 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":22}
I20260926 06:15:02.289876 4125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:02.286888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.193:36493 (local address 127.2.110.195:45167)
0926 06:15:02.287047 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:02.287058 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:02.287126 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:02.287280 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:02.287281 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:02.287282 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:15:02.287372 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:02.287374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.288219 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.288220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:02.289004 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:02.289006 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.289582 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.289583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:02.289670 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:02.289675 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 06:15:02.289730 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 06:15:02.289768 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":20,"thread_start_us":90,"threads_started":1}
I20260926 06:15:02.290234 3902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32f72973923a4d32b640b85aa5bbe0b4" candidate_uuid: "a2a43bf937df4caf8fa833e7f9069167" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e758657151684f7bac40258b0e41a0c7" is_pre_election: true
I20260926 06:15:02.290236 3767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32f72973923a4d32b640b85aa5bbe0b4" candidate_uuid: "a2a43bf937df4caf8fa833e7f9069167" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9895ecbf03947dc8d50afde1f9f9a1e" is_pre_election: true
I20260926 06:15:02.290373 3902 raft_consensus.cc:2454] T 32f72973923a4d32b640b85aa5bbe0b4 P e758657151684f7bac40258b0e41a0c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a43bf937df4caf8fa833e7f9069167 in term 0.
I20260926 06:15:02.290373 3767 raft_consensus.cc:2454] T 32f72973923a4d32b640b85aa5bbe0b4 P f9895ecbf03947dc8d50afde1f9f9a1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a43bf937df4caf8fa833e7f9069167 in term 0.
I20260926 06:15:02.290570 3978 leader_election.cc:302] T 32f72973923a4d32b640b85aa5bbe0b4 P a2a43bf937df4caf8fa833e7f9069167 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a43bf937df4caf8fa833e7f9069167, e758657151684f7bac40258b0e41a0c7; no voters:
I20260926 06:15:02.290684 4123 raft_consensus.cc:2781] T 32f72973923a4d32b640b85aa5bbe0b4 P a2a43bf937df4caf8fa833e7f9069167 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:15:02.290787 4123 raft_consensus.cc:482] T 32f72973923a4d32b640b85aa5bbe0b4 P a2a43bf937df4caf8fa833e7f9069167 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:15:02.290822 4123 raft_consensus.cc:3037] T 32f72973923a4d32b640b85aa5bbe0b4 P a2a43bf937df4caf8fa833e7f9069167 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:02.291730 4123 raft_consensus.cc:504] T 32f72973923a4d32b640b85aa5bbe0b4 P a2a43bf937df4caf8fa833e7f9069167 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a43bf937df4caf8fa833e7f9069167" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 40247 } } peers { permanent_uuid: "f9895ecbf03947dc8d50afde1f9f9a1e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 36493 } } peers { permanent_uuid: "e758657151684f7bac40258b0e41a0c7" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36241 } }
I20260926 06:15:02.291977 3767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32f72973923a4d32b640b85aa5bbe0b4" candidate_uuid: "a2a43bf937df4caf8fa833e7f9069167" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9895ecbf03947dc8d50afde1f9f9a1e"
I20260926 06:15:02.292052 3767 raft_consensus.cc:3037] T 32f72973923a4d32b640b85aa5bbe0b4 P f9895ecbf03947dc8d50afde1f9f9a1e [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:02.292783 3767 raft_consensus.cc:2454] T 32f72973923a4d32b640b85aa5bbe0b4 P f9895ecbf03947dc8d50afde1f9f9a1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a43bf937df4caf8fa833e7f9069167 in term 1.
I20260926 06:15:02.292914 4123 leader_election.cc:288] T 32f72973923a4d32b640b85aa5bbe0b4 P a2a43bf937df4caf8fa833e7f9069167 [CANDIDATE]: Term 1 election: Requested vote from peers f9895ecbf03947dc8d50afde1f9f9a1e (127.2.110.193:36493), e758657151684f7bac40258b0e41a0c7 (127.2.110.194:36241)
I20260926 06:15:02.293080 3902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32f72973923a4d32b640b85aa5bbe0b4" candidate_uuid: "a2a43bf937df4caf8fa833e7f9069167" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e758657151684f7bac40258b0e41a0c7"
I20260926 06:15:02.293161 3902 raft_consensus.cc:3037] T 32f72973923a4d32b640b85aa5bbe0b4 P e758657151684f7bac40258b0e41a0c7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:02.293311 3980 leader_election.cc:302] T 32f72973923a4d32b640b85aa5bbe0b4 P a2a43bf937df4caf8fa833e7f9069167 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a43bf937df4caf8fa833e7f9069167, f9895ecbf03947dc8d50afde1f9f9a1e; no voters:
I20260926 06:15:02.293438 4123 raft_consensus.cc:2781] T 32f72973923a4d32b640b85aa5bbe0b4 P a2a43bf937df4caf8fa833e7f9069167 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:15:02.293607 4123 raft_consensus.cc:686] T 32f72973923a4d32b640b85aa5bbe0b4 P a2a43bf937df4caf8fa833e7f9069167 [term 1 LEADER]: Becoming Leader. State: Replica: a2a43bf937df4caf8fa833e7f9069167, State: Running, Role: LEADER
I20260926 06:15:02.293707 4123 consensus_queue.cc:237] T 32f72973923a4d32b640b85aa5bbe0b4 P a2a43bf937df4caf8fa833e7f9069167 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a43bf937df4caf8fa833e7f9069167" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 40247 } } peers { permanent_uuid: "f9895ecbf03947dc8d50afde1f9f9a1e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 36493 } } peers { permanent_uuid: "e758657151684f7bac40258b0e41a0c7" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36241 } }
I20260926 06:15:02.294047 3902 raft_consensus.cc:2454] T 32f72973923a4d32b640b85aa5bbe0b4 P e758657151684f7bac40258b0e41a0c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a43bf937df4caf8fa833e7f9069167 in term 1.
I20260926 06:15:02.294555 3609 catalog_manager.cc:5495] T 32f72973923a4d32b640b85aa5bbe0b4 P a2a43bf937df4caf8fa833e7f9069167 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2a43bf937df4caf8fa833e7f9069167 (127.2.110.195). New cstate: current_term: 1 leader_uuid: "a2a43bf937df4caf8fa833e7f9069167" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a43bf937df4caf8fa833e7f9069167" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 40247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9895ecbf03947dc8d50afde1f9f9a1e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 36493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e758657151684f7bac40258b0e41a0c7" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36241 } health_report { overall_health: UNKNOWN } } }
I20260926 06:15:02.304214 4113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:02.301525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54344 (local address 127.2.110.195:40247)
0926 06:15:02.301575 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:02.301577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:02.301609 (+ 32us) server_negotiation.cc:407] Connection header received
0926 06:15:02.301697 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:02.301698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:02.301714 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:02.301771 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:15:02.301922 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.302394 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.303183 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.303378 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.303441 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:02.303832 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:15:02.303943 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:15:02.304008 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:02.304062 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 06:15:02.304085 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 06:15:02.306825 3767 raft_consensus.cc:1261] T 32f72973923a4d32b640b85aa5bbe0b4 P f9895ecbf03947dc8d50afde1f9f9a1e [term 1 FOLLOWER]: Refusing update from remote peer a2a43bf937df4caf8fa833e7f9069167: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:15:02.306923 3902 raft_consensus.cc:1261] T 32f72973923a4d32b640b85aa5bbe0b4 P e758657151684f7bac40258b0e41a0c7 [term 1 FOLLOWER]: Refusing update from remote peer a2a43bf937df4caf8fa833e7f9069167: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:15:02.307228 4126 consensus_queue.cc:1035] T 32f72973923a4d32b640b85aa5bbe0b4 P a2a43bf937df4caf8fa833e7f9069167 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9895ecbf03947dc8d50afde1f9f9a1e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 36493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:15:02.307452 4123 consensus_queue.cc:1035] T 32f72973923a4d32b640b85aa5bbe0b4 P a2a43bf937df4caf8fa833e7f9069167 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e758657151684f7bac40258b0e41a0c7" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:15:02.309976 4129 mvcc.cc:205] Tried to move back new op lower bound from 7333491926244941824 to 7333491926195769344. Current Snapshot: MvccSnapshot[applied={T|T < 7333491926244941824}]
I20260926 06:15:02.310259 4130 mvcc.cc:205] Tried to move back new op lower bound from 7333491926244941824 to 7333491926195769344. Current Snapshot: MvccSnapshot[applied={T|T < 7333491926244941824}]
I20260926 06:15:02.310947 4131 mvcc.cc:205] Tried to move back new op lower bound from 7333491926244941824 to 7333491926195769344. Current Snapshot: MvccSnapshot[applied={T|T < 7333491926244941824}]
I20260926 06:15:02.313455 3610 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:15:02.313560 3610 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:15:02.314347 3610 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1eff30d645784947bf39eafdb0868de2: Sending DeleteTablet for 3 replicas of tablet 32f72973923a4d32b640b85aa5bbe0b4
I20260926 06:15:02.314702 4027 tablet_service.cc:1514] Processing DeleteTablet for tablet 32f72973923a4d32b640b85aa5bbe0b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:15:02 UTC) from {username='kudu', principal='kudu/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:54332
I20260926 06:15:02.314699 3887 tablet_service.cc:1514] Processing DeleteTablet for tablet 32f72973923a4d32b640b85aa5bbe0b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:15:02 UTC) from {username='kudu', principal='kudu/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:60726
I20260926 06:15:02.314944 3743 tablet_service.cc:1514] Processing DeleteTablet for tablet 32f72973923a4d32b640b85aa5bbe0b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:15:02 UTC) from {username='kudu', principal='kudu/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:56140
I20260926 06:15:02.314966 3610 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51166:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:15:02.315130 3610 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:15:02.315304 4139 tablet_replica.cc:330] T 32f72973923a4d32b640b85aa5bbe0b4 P e758657151684f7bac40258b0e41a0c7: stopping tablet replica
I20260926 06:15:02.315352 4140 tablet_replica.cc:330] T 32f72973923a4d32b640b85aa5bbe0b4 P f9895ecbf03947dc8d50afde1f9f9a1e: stopping tablet replica
I20260926 06:15:02.315457 4140 raft_consensus.cc:2229] T 32f72973923a4d32b640b85aa5bbe0b4 P f9895ecbf03947dc8d50afde1f9f9a1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:15:02.315476 4138 tablet_replica.cc:330] T 32f72973923a4d32b640b85aa5bbe0b4 P a2a43bf937df4caf8fa833e7f9069167: stopping tablet replica
I20260926 06:15:02.315585 4140 raft_consensus.cc:2258] T 32f72973923a4d32b640b85aa5bbe0b4 P f9895ecbf03947dc8d50afde1f9f9a1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:15:02.315729 4139 raft_consensus.cc:2229] T 32f72973923a4d32b640b85aa5bbe0b4 P e758657151684f7bac40258b0e41a0c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:15:02.315850 4138 raft_consensus.cc:2229] T 32f72973923a4d32b640b85aa5bbe0b4 P a2a43bf937df4caf8fa833e7f9069167 [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:15:02.316036 4138 raft_consensus.cc:2258] T 32f72973923a4d32b640b85aa5bbe0b4 P a2a43bf937df4caf8fa833e7f9069167 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:15:02.316192 4139 raft_consensus.cc:2258] T 32f72973923a4d32b640b85aa5bbe0b4 P e758657151684f7bac40258b0e41a0c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:15:02.316497 4140 ts_tablet_manager.cc:1905] T 32f72973923a4d32b640b85aa5bbe0b4 P f9895ecbf03947dc8d50afde1f9f9a1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:15:02.316898 4027 tablet_service.cc:1467] Processing CreateTablet for tablet b5aed8266c9d4eaea0201b4ef8ccb882 (DEFAULT_TABLE table=test-table [id=05dd0089c2c247989702c11b798dfd62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:15:02.316898 3887 tablet_service.cc:1467] Processing CreateTablet for tablet b5aed8266c9d4eaea0201b4ef8ccb882 (DEFAULT_TABLE table=test-table [id=05dd0089c2c247989702c11b798dfd62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:15:02.317009 3887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5aed8266c9d4eaea0201b4ef8ccb882. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:02.317009 4027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5aed8266c9d4eaea0201b4ef8ccb882. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:02.317178 3743 tablet_service.cc:1467] Processing CreateTablet for tablet b5aed8266c9d4eaea0201b4ef8ccb882 (DEFAULT_TABLE table=test-table [id=05dd0089c2c247989702c11b798dfd62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:15:02.317291 3743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5aed8266c9d4eaea0201b4ef8ccb882. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:02.318452 4116 tablet_bootstrap.cc:492] T b5aed8266c9d4eaea0201b4ef8ccb882 P a2a43bf937df4caf8fa833e7f9069167: Bootstrap starting.
I20260926 06:15:02.318876 4116 tablet_bootstrap.cc:654] T b5aed8266c9d4eaea0201b4ef8ccb882 P a2a43bf937df4caf8fa833e7f9069167: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:02.319075 4118 tablet_bootstrap.cc:492] T b5aed8266c9d4eaea0201b4ef8ccb882 P e758657151684f7bac40258b0e41a0c7: Bootstrap starting.
I20260926 06:15:02.319088 4117 tablet_bootstrap.cc:492] T b5aed8266c9d4eaea0201b4ef8ccb882 P f9895ecbf03947dc8d50afde1f9f9a1e: Bootstrap starting.
I20260926 06:15:02.319512 4118 tablet_bootstrap.cc:654] T b5aed8266c9d4eaea0201b4ef8ccb882 P e758657151684f7bac40258b0e41a0c7: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:02.319525 4117 tablet_bootstrap.cc:654] T b5aed8266c9d4eaea0201b4ef8ccb882 P f9895ecbf03947dc8d50afde1f9f9a1e: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:02.319989 4139 ts_tablet_manager.cc:1905] T 32f72973923a4d32b640b85aa5bbe0b4 P e758657151684f7bac40258b0e41a0c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:15:02.320251 4138 ts_tablet_manager.cc:1905] T 32f72973923a4d32b640b85aa5bbe0b4 P a2a43bf937df4caf8fa833e7f9069167: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:15:02.321743 4117 tablet_bootstrap.cc:492] T b5aed8266c9d4eaea0201b4ef8ccb882 P f9895ecbf03947dc8d50afde1f9f9a1e: No bootstrap required, opened a new log
I20260926 06:15:02.321802 4117 ts_tablet_manager.cc:1397] T b5aed8266c9d4eaea0201b4ef8ccb882 P f9895ecbf03947dc8d50afde1f9f9a1e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 06:15:02.321908 4118 tablet_bootstrap.cc:492] T b5aed8266c9d4eaea0201b4ef8ccb882 P e758657151684f7bac40258b0e41a0c7: No bootstrap required, opened a new log
I20260926 06:15:02.321965 4118 ts_tablet_manager.cc:1397] T b5aed8266c9d4eaea0201b4ef8ccb882 P e758657151684f7bac40258b0e41a0c7: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260926 06:15:02.321971 4117 raft_consensus.cc:348] T b5aed8266c9d4eaea0201b4ef8ccb882 P f9895ecbf03947dc8d50afde1f9f9a1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9895ecbf03947dc8d50afde1f9f9a1e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 36493 } } peers { permanent_uuid: "e758657151684f7bac40258b0e41a0c7" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36241 } } peers { permanent_uuid: "a2a43bf937df4caf8fa833e7f9069167" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 40247 } }
I20260926 06:15:02.322053 4117 raft_consensus.cc:374] T b5aed8266c9d4eaea0201b4ef8ccb882 P f9895ecbf03947dc8d50afde1f9f9a1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:02.322080 4117 raft_consensus.cc:729] T b5aed8266c9d4eaea0201b4ef8ccb882 P f9895ecbf03947dc8d50afde1f9f9a1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9895ecbf03947dc8d50afde1f9f9a1e, State: Initialized, Role: FOLLOWER
I20260926 06:15:02.322144 4117 consensus_queue.cc:260] T b5aed8266c9d4eaea0201b4ef8ccb882 P f9895ecbf03947dc8d50afde1f9f9a1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9895ecbf03947dc8d50afde1f9f9a1e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 36493 } } peers { permanent_uuid: "e758657151684f7bac40258b0e41a0c7" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36241 } } peers { permanent_uuid: "a2a43bf937df4caf8fa833e7f9069167" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 40247 } }
I20260926 06:15:02.322265 4117 ts_tablet_manager.cc:1428] T b5aed8266c9d4eaea0201b4ef8ccb882 P f9895ecbf03947dc8d50afde1f9f9a1e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 06:15:02.322579 4118 raft_consensus.cc:348] T b5aed8266c9d4eaea0201b4ef8ccb882 P e758657151684f7bac40258b0e41a0c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9895ecbf03947dc8d50afde1f9f9a1e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 36493 } } peers { permanent_uuid: "e758657151684f7bac40258b0e41a0c7" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36241 } } peers { permanent_uuid: "a2a43bf937df4caf8fa833e7f9069167" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 40247 } }
I20260926 06:15:02.322665 4118 raft_consensus.cc:374] T b5aed8266c9d4eaea0201b4ef8ccb882 P e758657151684f7bac40258b0e41a0c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:02.322687 4118 raft_consensus.cc:729] T b5aed8266c9d4eaea0201b4ef8ccb882 P e758657151684f7bac40258b0e41a0c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e758657151684f7bac40258b0e41a0c7, State: Initialized, Role: FOLLOWER
I20260926 06:15:02.322741 4118 consensus_queue.cc:260] T b5aed8266c9d4eaea0201b4ef8ccb882 P e758657151684f7bac40258b0e41a0c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9895ecbf03947dc8d50afde1f9f9a1e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 36493 } } peers { permanent_uuid: "e758657151684f7bac40258b0e41a0c7" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36241 } } peers { permanent_uuid: "a2a43bf937df4caf8fa833e7f9069167" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 40247 } }
I20260926 06:15:02.322860 4118 ts_tablet_manager.cc:1428] T b5aed8266c9d4eaea0201b4ef8ccb882 P e758657151684f7bac40258b0e41a0c7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 06:15:02.323122 4140 ts_tablet_manager.cc:1918] T 32f72973923a4d32b640b85aa5bbe0b4 P f9895ecbf03947dc8d50afde1f9f9a1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:15:02.323220 4140 log.cc:1196] T 32f72973923a4d32b640b85aa5bbe0b4 P f9895ecbf03947dc8d50afde1f9f9a1e: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/ts-0/wal/wals/32f72973923a4d32b640b85aa5bbe0b4
I20260926 06:15:02.323515 4140 ts_tablet_manager.cc:1939] T 32f72973923a4d32b640b85aa5bbe0b4 P f9895ecbf03947dc8d50afde1f9f9a1e: Deleting consensus metadata
I20260926 06:15:02.323999 3596 catalog_manager.cc:4841] TS f9895ecbf03947dc8d50afde1f9f9a1e (127.2.110.193:36493): tablet 32f72973923a4d32b640b85aa5bbe0b4 (table test-table [id=44d36946593e4c0e823d222ca40b8011]) successfully deleted
I20260926 06:15:02.324626 4139 ts_tablet_manager.cc:1918] T 32f72973923a4d32b640b85aa5bbe0b4 P e758657151684f7bac40258b0e41a0c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:15:02.324729 4139 log.cc:1196] T 32f72973923a4d32b640b85aa5bbe0b4 P e758657151684f7bac40258b0e41a0c7: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/ts-1/wal/wals/32f72973923a4d32b640b85aa5bbe0b4
I20260926 06:15:02.324955 4139 ts_tablet_manager.cc:1939] T 32f72973923a4d32b640b85aa5bbe0b4 P e758657151684f7bac40258b0e41a0c7: Deleting consensus metadata
I20260926 06:15:02.325521 3594 catalog_manager.cc:4841] TS e758657151684f7bac40258b0e41a0c7 (127.2.110.194:36241): tablet 32f72973923a4d32b640b85aa5bbe0b4 (table test-table [id=44d36946593e4c0e823d222ca40b8011]) successfully deleted
I20260926 06:15:02.325555 4138 ts_tablet_manager.cc:1918] T 32f72973923a4d32b640b85aa5bbe0b4 P a2a43bf937df4caf8fa833e7f9069167: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:15:02.325625 4138 log.cc:1196] T 32f72973923a4d32b640b85aa5bbe0b4 P a2a43bf937df4caf8fa833e7f9069167: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/ts-2/wal/wals/32f72973923a4d32b640b85aa5bbe0b4
I20260926 06:15:02.326004 4138 ts_tablet_manager.cc:1939] T 32f72973923a4d32b640b85aa5bbe0b4 P a2a43bf937df4caf8fa833e7f9069167: Deleting consensus metadata
I20260926 06:15:02.326418 3595 catalog_manager.cc:4841] TS a2a43bf937df4caf8fa833e7f9069167 (127.2.110.195:40247): tablet 32f72973923a4d32b640b85aa5bbe0b4 (table test-table [id=44d36946593e4c0e823d222ca40b8011]) successfully deleted
I20260926 06:15:02.326682 4116 tablet_bootstrap.cc:492] T b5aed8266c9d4eaea0201b4ef8ccb882 P a2a43bf937df4caf8fa833e7f9069167: No bootstrap required, opened a new log
I20260926 06:15:02.326728 4116 ts_tablet_manager.cc:1397] T b5aed8266c9d4eaea0201b4ef8ccb882 P a2a43bf937df4caf8fa833e7f9069167: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s
I20260926 06:15:02.326902 4116 raft_consensus.cc:348] T b5aed8266c9d4eaea0201b4ef8ccb882 P a2a43bf937df4caf8fa833e7f9069167 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9895ecbf03947dc8d50afde1f9f9a1e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 36493 } } peers { permanent_uuid: "e758657151684f7bac40258b0e41a0c7" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36241 } } peers { permanent_uuid: "a2a43bf937df4caf8fa833e7f9069167" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 40247 } }
I20260926 06:15:02.326967 4116 raft_consensus.cc:374] T b5aed8266c9d4eaea0201b4ef8ccb882 P a2a43bf937df4caf8fa833e7f9069167 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:02.326980 4116 raft_consensus.cc:729] T b5aed8266c9d4eaea0201b4ef8ccb882 P a2a43bf937df4caf8fa833e7f9069167 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a43bf937df4caf8fa833e7f9069167, State: Initialized, Role: FOLLOWER
I20260926 06:15:02.327031 4116 consensus_queue.cc:260] T b5aed8266c9d4eaea0201b4ef8ccb882 P a2a43bf937df4caf8fa833e7f9069167 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9895ecbf03947dc8d50afde1f9f9a1e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 36493 } } peers { permanent_uuid: "e758657151684f7bac40258b0e41a0c7" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36241 } } peers { permanent_uuid: "a2a43bf937df4caf8fa833e7f9069167" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 40247 } }
I20260926 06:15:02.327136 4116 ts_tablet_manager.cc:1428] T b5aed8266c9d4eaea0201b4ef8ccb882 P a2a43bf937df4caf8fa833e7f9069167: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260926 06:15:02.401907 3814 tablet.cc:2369] T b5aed8266c9d4eaea0201b4ef8ccb882 P f9895ecbf03947dc8d50afde1f9f9a1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 06:15:02.482820 4094 tablet.cc:2369] T b5aed8266c9d4eaea0201b4ef8ccb882 P a2a43bf937df4caf8fa833e7f9069167: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 06:15:02.566771 3954 tablet.cc:2369] T b5aed8266c9d4eaea0201b4ef8ccb882 P e758657151684f7bac40258b0e41a0c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:15:02.593662 4122 raft_consensus.cc:482] T b5aed8266c9d4eaea0201b4ef8ccb882 P e758657151684f7bac40258b0e41a0c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:15:02.593784 4122 raft_consensus.cc:504] T b5aed8266c9d4eaea0201b4ef8ccb882 P e758657151684f7bac40258b0e41a0c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9895ecbf03947dc8d50afde1f9f9a1e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 36493 } } peers { permanent_uuid: "e758657151684f7bac40258b0e41a0c7" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36241 } } peers { permanent_uuid: "a2a43bf937df4caf8fa833e7f9069167" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 40247 } }
I20260926 06:15:02.594420 4122 leader_election.cc:288] T b5aed8266c9d4eaea0201b4ef8ccb882 P e758657151684f7bac40258b0e41a0c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9895ecbf03947dc8d50afde1f9f9a1e (127.2.110.193:36493), a2a43bf937df4caf8fa833e7f9069167 (127.2.110.195:40247)
I20260926 06:15:02.598799 4113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:02.594801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:41211 (local address 127.2.110.195:40247)
0926 06:15:02.594850 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:02.594853 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:02.594866 (+ 13us) server_negotiation.cc:407] Connection header received
0926 06:15:02.594950 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:02.594951 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:02.594963 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:02.595033 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:15:02.595348 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.596103 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.597825 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.598406 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.598479 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:02.598488 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:02.598673 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 06:15:02.598718 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 06:15:02.598799 4115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:02.594809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:57679 (local address 127.2.110.193:36493)
0926 06:15:02.594873 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:02.594875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:02.594883 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:15:02.594959 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:02.594960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:02.594970 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:02.595007 (+ 37us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:15:02.595346 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.596103 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.597825 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.598392 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.598479 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:02.598489 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:02.598673 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 06:15:02.598727 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 06:15:02.598834 4145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:02.594671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.195:40247 (local address 127.2.110.194:41211)
0926 06:15:02.594818 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:02.594827 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:02.594869 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:02.595098 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:02.595100 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:02.595105 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:15:02.595248 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:02.595249 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.596252 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.596254 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:02.597720 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:02.597723 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.598496 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.598497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:02.598585 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:02.598591 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 06:15:02.598656 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 06:15:02.598720 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":98,"thread_start_us":64,"threads_started":1}
I20260926 06:15:02.598838 4144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:02.594670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.193:36493 (local address 127.2.110.194:57679)
0926 06:15:02.594786 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:02.594799 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:02.594866 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:02.595115 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:02.595116 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:02.595117 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:15:02.595246 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:02.595249 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.596252 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.596260 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:02.597720 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:02.597723 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.598498 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.598498 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:02.598583 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:02.598592 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 06:15:02.598656 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 06:15:02.598719 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":120,"thread_start_us":67,"threads_started":1}
I20260926 06:15:02.599182 4047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5aed8266c9d4eaea0201b4ef8ccb882" candidate_uuid: "e758657151684f7bac40258b0e41a0c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a43bf937df4caf8fa833e7f9069167" is_pre_election: true
I20260926 06:15:02.599323 3767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5aed8266c9d4eaea0201b4ef8ccb882" candidate_uuid: "e758657151684f7bac40258b0e41a0c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9895ecbf03947dc8d50afde1f9f9a1e" is_pre_election: true
I20260926 06:15:02.599378 4047 raft_consensus.cc:2454] T b5aed8266c9d4eaea0201b4ef8ccb882 P a2a43bf937df4caf8fa833e7f9069167 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e758657151684f7bac40258b0e41a0c7 in term 0.
I20260926 06:15:02.599406 3767 raft_consensus.cc:2454] T b5aed8266c9d4eaea0201b4ef8ccb882 P f9895ecbf03947dc8d50afde1f9f9a1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e758657151684f7bac40258b0e41a0c7 in term 0.
I20260926 06:15:02.599582 3840 leader_election.cc:302] T b5aed8266c9d4eaea0201b4ef8ccb882 P e758657151684f7bac40258b0e41a0c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e758657151684f7bac40258b0e41a0c7, f9895ecbf03947dc8d50afde1f9f9a1e; no voters:
I20260926 06:15:02.599725 4122 raft_consensus.cc:2781] T b5aed8266c9d4eaea0201b4ef8ccb882 P e758657151684f7bac40258b0e41a0c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:15:02.599776 4122 raft_consensus.cc:482] T b5aed8266c9d4eaea0201b4ef8ccb882 P e758657151684f7bac40258b0e41a0c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:15:02.599805 4122 raft_consensus.cc:3037] T b5aed8266c9d4eaea0201b4ef8ccb882 P e758657151684f7bac40258b0e41a0c7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:02.600543 4122 raft_consensus.cc:504] T b5aed8266c9d4eaea0201b4ef8ccb882 P e758657151684f7bac40258b0e41a0c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9895ecbf03947dc8d50afde1f9f9a1e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 36493 } } peers { permanent_uuid: "e758657151684f7bac40258b0e41a0c7" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36241 } } peers { permanent_uuid: "a2a43bf937df4caf8fa833e7f9069167" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 40247 } }
I20260926 06:15:02.600708 4122 leader_election.cc:288] T b5aed8266c9d4eaea0201b4ef8ccb882 P e758657151684f7bac40258b0e41a0c7 [CANDIDATE]: Term 1 election: Requested vote from peers f9895ecbf03947dc8d50afde1f9f9a1e (127.2.110.193:36493), a2a43bf937df4caf8fa833e7f9069167 (127.2.110.195:40247)
I20260926 06:15:02.600857 3767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5aed8266c9d4eaea0201b4ef8ccb882" candidate_uuid: "e758657151684f7bac40258b0e41a0c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9895ecbf03947dc8d50afde1f9f9a1e"
I20260926 06:15:02.600890 4047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5aed8266c9d4eaea0201b4ef8ccb882" candidate_uuid: "e758657151684f7bac40258b0e41a0c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a43bf937df4caf8fa833e7f9069167"
I20260926 06:15:02.600953 3767 raft_consensus.cc:3037] T b5aed8266c9d4eaea0201b4ef8ccb882 P f9895ecbf03947dc8d50afde1f9f9a1e [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:02.600966 4047 raft_consensus.cc:3037] T b5aed8266c9d4eaea0201b4ef8ccb882 P a2a43bf937df4caf8fa833e7f9069167 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:02.601680 3767 raft_consensus.cc:2454] T b5aed8266c9d4eaea0201b4ef8ccb882 P f9895ecbf03947dc8d50afde1f9f9a1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e758657151684f7bac40258b0e41a0c7 in term 1.
I20260926 06:15:02.601897 4047 raft_consensus.cc:2454] T b5aed8266c9d4eaea0201b4ef8ccb882 P a2a43bf937df4caf8fa833e7f9069167 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e758657151684f7bac40258b0e41a0c7 in term 1.
I20260926 06:15:02.602073 3840 leader_election.cc:302] T b5aed8266c9d4eaea0201b4ef8ccb882 P e758657151684f7bac40258b0e41a0c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e758657151684f7bac40258b0e41a0c7, f9895ecbf03947dc8d50afde1f9f9a1e; no voters:
I20260926 06:15:02.602180 4122 raft_consensus.cc:2781] T b5aed8266c9d4eaea0201b4ef8ccb882 P e758657151684f7bac40258b0e41a0c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:15:02.602288 4122 raft_consensus.cc:686] T b5aed8266c9d4eaea0201b4ef8ccb882 P e758657151684f7bac40258b0e41a0c7 [term 1 LEADER]: Becoming Leader. State: Replica: e758657151684f7bac40258b0e41a0c7, State: Running, Role: LEADER
I20260926 06:15:02.602372 4122 consensus_queue.cc:237] T b5aed8266c9d4eaea0201b4ef8ccb882 P e758657151684f7bac40258b0e41a0c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9895ecbf03947dc8d50afde1f9f9a1e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 36493 } } peers { permanent_uuid: "e758657151684f7bac40258b0e41a0c7" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36241 } } peers { permanent_uuid: "a2a43bf937df4caf8fa833e7f9069167" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 40247 } }
I20260926 06:15:02.603049 3609 catalog_manager.cc:5495] T b5aed8266c9d4eaea0201b4ef8ccb882 P e758657151684f7bac40258b0e41a0c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e758657151684f7bac40258b0e41a0c7 (127.2.110.194). New cstate: current_term: 1 leader_uuid: "e758657151684f7bac40258b0e41a0c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9895ecbf03947dc8d50afde1f9f9a1e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 36493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e758657151684f7bac40258b0e41a0c7" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36241 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a43bf937df4caf8fa833e7f9069167" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 40247 } health_report { overall_health: UNKNOWN } } }
Sep 26 06:15:02 dist-test-slave-hndd krb5kdc[3565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403301, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.254@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:15:02.701586 3660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:02.694230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51176 (local address 127.2.110.254:34967)
0926 06:15:02.694336 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:02.694339 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:02.694433 (+ 94us) server_negotiation.cc:407] Connection header received
0926 06:15:02.694525 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:02.694526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:02.694537 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:02.694615 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:02.695076 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.695578 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.696289 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.696473 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.696557 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:02.698702 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:02.698704 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:02.698704 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:02.698710 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:02.699952 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:02.700616 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:02.700617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:02.700617 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:02.700686 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:02.700927 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:02.700927 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:02.700928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:02.701038 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:02.701110 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:02.701423 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 06:15:02.701515 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 06:15:02.701741 4153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:02.694136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34967 (local address 127.0.0.1:51176)
0926 06:15:02.694348 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:02.694360 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:02.694456 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:02.694626 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:02.694627 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:02.694945 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:02.694989 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:02.694991 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.695681 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.695682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:02.696203 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:02.696205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.696557 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.696558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:02.696616 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:02.697150 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:02.697154 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:02.698608 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:02.700120 (+ 1512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:02.700137 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:02.700140 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:02.700473 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:02.700787 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:02.700789 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:02.700789 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:02.700864 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:02.701166 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:02.701174 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:02.701242 (+ 68us) client_negotiation.cc:769] Sending connection context
0926 06:15:02.701405 (+ 163us) client_negotiation.cc:240] Negotiation successful
0926 06:15:02.701610 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":40,"thread_start_us":99,"threads_started":1}
I20260926 06:15:02.704015 3609 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:51176:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 06:15:02.704218 3609 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:15:02.705952 4027 tablet_service.cc:1467] Processing CreateTablet for tablet feff5d83990f48a0a3ddb648e03769fb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0e46a8c6f1e74fbf8bd2b1c7d51c7e58]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 06:15:02.705991 3887 tablet_service.cc:1467] Processing CreateTablet for tablet feff5d83990f48a0a3ddb648e03769fb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0e46a8c6f1e74fbf8bd2b1c7d51c7e58]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 06:15:02.706287 3887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet feff5d83990f48a0a3ddb648e03769fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:02.706287 4027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet feff5d83990f48a0a3ddb648e03769fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:02.706247 3743 tablet_service.cc:1467] Processing CreateTablet for tablet feff5d83990f48a0a3ddb648e03769fb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0e46a8c6f1e74fbf8bd2b1c7d51c7e58]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 06:15:02.706437 3743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet feff5d83990f48a0a3ddb648e03769fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:02.708192 4117 tablet_bootstrap.cc:492] T feff5d83990f48a0a3ddb648e03769fb P f9895ecbf03947dc8d50afde1f9f9a1e: Bootstrap starting.
I20260926 06:15:02.708236 4116 tablet_bootstrap.cc:492] T feff5d83990f48a0a3ddb648e03769fb P a2a43bf937df4caf8fa833e7f9069167: Bootstrap starting.
I20260926 06:15:02.708263 4118 tablet_bootstrap.cc:492] T feff5d83990f48a0a3ddb648e03769fb P e758657151684f7bac40258b0e41a0c7: Bootstrap starting.
I20260926 06:15:02.708616 4116 tablet_bootstrap.cc:654] T feff5d83990f48a0a3ddb648e03769fb P a2a43bf937df4caf8fa833e7f9069167: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:02.708678 4118 tablet_bootstrap.cc:654] T feff5d83990f48a0a3ddb648e03769fb P e758657151684f7bac40258b0e41a0c7: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:02.708678 4117 tablet_bootstrap.cc:654] T feff5d83990f48a0a3ddb648e03769fb P f9895ecbf03947dc8d50afde1f9f9a1e: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:02.709444 4118 tablet_bootstrap.cc:492] T feff5d83990f48a0a3ddb648e03769fb P e758657151684f7bac40258b0e41a0c7: No bootstrap required, opened a new log
I20260926 06:15:02.709501 4118 ts_tablet_manager.cc:1397] T feff5d83990f48a0a3ddb648e03769fb P e758657151684f7bac40258b0e41a0c7: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 06:15:02.709540 4117 tablet_bootstrap.cc:492] T feff5d83990f48a0a3ddb648e03769fb P f9895ecbf03947dc8d50afde1f9f9a1e: No bootstrap required, opened a new log
I20260926 06:15:02.709585 4117 ts_tablet_manager.cc:1397] T feff5d83990f48a0a3ddb648e03769fb P f9895ecbf03947dc8d50afde1f9f9a1e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:02.709656 4118 raft_consensus.cc:348] T feff5d83990f48a0a3ddb648e03769fb P e758657151684f7bac40258b0e41a0c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9895ecbf03947dc8d50afde1f9f9a1e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 36493 } } peers { permanent_uuid: "e758657151684f7bac40258b0e41a0c7" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36241 } } peers { permanent_uuid: "a2a43bf937df4caf8fa833e7f9069167" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 40247 } }
I20260926 06:15:02.709725 4118 raft_consensus.cc:374] T feff5d83990f48a0a3ddb648e03769fb P e758657151684f7bac40258b0e41a0c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:02.709749 4118 raft_consensus.cc:729] T feff5d83990f48a0a3ddb648e03769fb P e758657151684f7bac40258b0e41a0c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e758657151684f7bac40258b0e41a0c7, State: Initialized, Role: FOLLOWER
I20260926 06:15:02.709717 4117 raft_consensus.cc:348] T feff5d83990f48a0a3ddb648e03769fb P f9895ecbf03947dc8d50afde1f9f9a1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9895ecbf03947dc8d50afde1f9f9a1e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 36493 } } peers { permanent_uuid: "e758657151684f7bac40258b0e41a0c7" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36241 } } peers { permanent_uuid: "a2a43bf937df4caf8fa833e7f9069167" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 40247 } }
I20260926 06:15:02.709800 4117 raft_consensus.cc:374] T feff5d83990f48a0a3ddb648e03769fb P f9895ecbf03947dc8d50afde1f9f9a1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:02.709823 4117 raft_consensus.cc:729] T feff5d83990f48a0a3ddb648e03769fb P f9895ecbf03947dc8d50afde1f9f9a1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9895ecbf03947dc8d50afde1f9f9a1e, State: Initialized, Role: FOLLOWER
I20260926 06:15:02.709803 4118 consensus_queue.cc:260] T feff5d83990f48a0a3ddb648e03769fb P e758657151684f7bac40258b0e41a0c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9895ecbf03947dc8d50afde1f9f9a1e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 36493 } } peers { permanent_uuid: "e758657151684f7bac40258b0e41a0c7" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36241 } } peers { permanent_uuid: "a2a43bf937df4caf8fa833e7f9069167" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 40247 } }
I20260926 06:15:02.709870 4117 consensus_queue.cc:260] T feff5d83990f48a0a3ddb648e03769fb P f9895ecbf03947dc8d50afde1f9f9a1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9895ecbf03947dc8d50afde1f9f9a1e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 36493 } } peers { permanent_uuid: "e758657151684f7bac40258b0e41a0c7" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36241 } } peers { permanent_uuid: "a2a43bf937df4caf8fa833e7f9069167" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 40247 } }
I20260926 06:15:02.709899 4118 ts_tablet_manager.cc:1428] T feff5d83990f48a0a3ddb648e03769fb P e758657151684f7bac40258b0e41a0c7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:02.709965 4117 ts_tablet_manager.cc:1428] T feff5d83990f48a0a3ddb648e03769fb P f9895ecbf03947dc8d50afde1f9f9a1e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:02.710043 4124 tablet_replica.cc:439] T feff5d83990f48a0a3ddb648e03769fb P f9895ecbf03947dc8d50afde1f9f9a1e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9895ecbf03947dc8d50afde1f9f9a1e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 36493 } } peers { permanent_uuid: "e758657151684f7bac40258b0e41a0c7" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36241 } } peers { permanent_uuid: "a2a43bf937df4caf8fa833e7f9069167" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 40247 } } }
I20260926 06:15:02.710163 4124 tablet_replica.cc:442] T feff5d83990f48a0a3ddb648e03769fb P f9895ecbf03947dc8d50afde1f9f9a1e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:15:02.710189 4116 tablet_bootstrap.cc:492] T feff5d83990f48a0a3ddb648e03769fb P a2a43bf937df4caf8fa833e7f9069167: No bootstrap required, opened a new log
I20260926 06:15:02.710245 4116 ts_tablet_manager.cc:1397] T feff5d83990f48a0a3ddb648e03769fb P a2a43bf937df4caf8fa833e7f9069167: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 06:15:02.710356 4116 raft_consensus.cc:348] T feff5d83990f48a0a3ddb648e03769fb P a2a43bf937df4caf8fa833e7f9069167 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9895ecbf03947dc8d50afde1f9f9a1e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 36493 } } peers { permanent_uuid: "e758657151684f7bac40258b0e41a0c7" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36241 } } peers { permanent_uuid: "a2a43bf937df4caf8fa833e7f9069167" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 40247 } }
I20260926 06:15:02.710384 4122 tablet_replica.cc:439] T feff5d83990f48a0a3ddb648e03769fb P e758657151684f7bac40258b0e41a0c7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9895ecbf03947dc8d50afde1f9f9a1e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 36493 } } peers { permanent_uuid: "e758657151684f7bac40258b0e41a0c7" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36241 } } peers { permanent_uuid: "a2a43bf937df4caf8fa833e7f9069167" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 40247 } } }
I20260926 06:15:02.710433 4116 raft_consensus.cc:374] T feff5d83990f48a0a3ddb648e03769fb P a2a43bf937df4caf8fa833e7f9069167 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:02.710455 4116 raft_consensus.cc:729] T feff5d83990f48a0a3ddb648e03769fb P a2a43bf937df4caf8fa833e7f9069167 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a43bf937df4caf8fa833e7f9069167, State: Initialized, Role: FOLLOWER
I20260926 06:15:02.710458 4122 tablet_replica.cc:442] T feff5d83990f48a0a3ddb648e03769fb P e758657151684f7bac40258b0e41a0c7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:15:02.710513 4116 consensus_queue.cc:260] T feff5d83990f48a0a3ddb648e03769fb P a2a43bf937df4caf8fa833e7f9069167 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9895ecbf03947dc8d50afde1f9f9a1e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 36493 } } peers { permanent_uuid: "e758657151684f7bac40258b0e41a0c7" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36241 } } peers { permanent_uuid: "a2a43bf937df4caf8fa833e7f9069167" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 40247 } }
I20260926 06:15:02.710605 4116 ts_tablet_manager.cc:1428] T feff5d83990f48a0a3ddb648e03769fb P a2a43bf937df4caf8fa833e7f9069167: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:02.710659 4126 tablet_replica.cc:439] T feff5d83990f48a0a3ddb648e03769fb P a2a43bf937df4caf8fa833e7f9069167: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9895ecbf03947dc8d50afde1f9f9a1e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 36493 } } peers { permanent_uuid: "e758657151684f7bac40258b0e41a0c7" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36241 } } peers { permanent_uuid: "a2a43bf937df4caf8fa833e7f9069167" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 40247 } } }
I20260926 06:15:02.710721 4126 tablet_replica.cc:442] T feff5d83990f48a0a3ddb648e03769fb P a2a43bf937df4caf8fa833e7f9069167: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:15:02.765548 4126 raft_consensus.cc:482] T feff5d83990f48a0a3ddb648e03769fb P a2a43bf937df4caf8fa833e7f9069167 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:15:02.765662 4126 raft_consensus.cc:504] T feff5d83990f48a0a3ddb648e03769fb P a2a43bf937df4caf8fa833e7f9069167 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9895ecbf03947dc8d50afde1f9f9a1e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 36493 } } peers { permanent_uuid: "e758657151684f7bac40258b0e41a0c7" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36241 } } peers { permanent_uuid: "a2a43bf937df4caf8fa833e7f9069167" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 40247 } }
I20260926 06:15:02.765889 4126 leader_election.cc:288] T feff5d83990f48a0a3ddb648e03769fb P a2a43bf937df4caf8fa833e7f9069167 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9895ecbf03947dc8d50afde1f9f9a1e (127.2.110.193:36493), e758657151684f7bac40258b0e41a0c7 (127.2.110.194:36241)
I20260926 06:15:02.766115 3902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feff5d83990f48a0a3ddb648e03769fb" candidate_uuid: "a2a43bf937df4caf8fa833e7f9069167" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e758657151684f7bac40258b0e41a0c7" is_pre_election: true
I20260926 06:15:02.766219 3902 raft_consensus.cc:2454] T feff5d83990f48a0a3ddb648e03769fb P e758657151684f7bac40258b0e41a0c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a43bf937df4caf8fa833e7f9069167 in term 0.
I20260926 06:15:02.766168 3767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feff5d83990f48a0a3ddb648e03769fb" candidate_uuid: "a2a43bf937df4caf8fa833e7f9069167" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9895ecbf03947dc8d50afde1f9f9a1e" is_pre_election: true
I20260926 06:15:02.766306 3767 raft_consensus.cc:2454] T feff5d83990f48a0a3ddb648e03769fb P f9895ecbf03947dc8d50afde1f9f9a1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a43bf937df4caf8fa833e7f9069167 in term 0.
I20260926 06:15:02.766381 3978 leader_election.cc:302] T feff5d83990f48a0a3ddb648e03769fb P a2a43bf937df4caf8fa833e7f9069167 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a43bf937df4caf8fa833e7f9069167, e758657151684f7bac40258b0e41a0c7; no voters:
I20260926 06:15:02.766461 4126 raft_consensus.cc:2781] T feff5d83990f48a0a3ddb648e03769fb P a2a43bf937df4caf8fa833e7f9069167 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:15:02.766510 4126 raft_consensus.cc:482] T feff5d83990f48a0a3ddb648e03769fb P a2a43bf937df4caf8fa833e7f9069167 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:15:02.766525 4126 raft_consensus.cc:3037] T feff5d83990f48a0a3ddb648e03769fb P a2a43bf937df4caf8fa833e7f9069167 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:02.767256 4126 raft_consensus.cc:504] T feff5d83990f48a0a3ddb648e03769fb P a2a43bf937df4caf8fa833e7f9069167 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9895ecbf03947dc8d50afde1f9f9a1e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 36493 } } peers { permanent_uuid: "e758657151684f7bac40258b0e41a0c7" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36241 } } peers { permanent_uuid: "a2a43bf937df4caf8fa833e7f9069167" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 40247 } }
I20260926 06:15:02.767403 4126 leader_election.cc:288] T feff5d83990f48a0a3ddb648e03769fb P a2a43bf937df4caf8fa833e7f9069167 [CANDIDATE]: Term 1 election: Requested vote from peers f9895ecbf03947dc8d50afde1f9f9a1e (127.2.110.193:36493), e758657151684f7bac40258b0e41a0c7 (127.2.110.194:36241)
I20260926 06:15:02.767590 3767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feff5d83990f48a0a3ddb648e03769fb" candidate_uuid: "a2a43bf937df4caf8fa833e7f9069167" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9895ecbf03947dc8d50afde1f9f9a1e"
I20260926 06:15:02.767617 3902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feff5d83990f48a0a3ddb648e03769fb" candidate_uuid: "a2a43bf937df4caf8fa833e7f9069167" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e758657151684f7bac40258b0e41a0c7"
I20260926 06:15:02.767686 3902 raft_consensus.cc:3037] T feff5d83990f48a0a3ddb648e03769fb P e758657151684f7bac40258b0e41a0c7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:02.767689 3767 raft_consensus.cc:3037] T feff5d83990f48a0a3ddb648e03769fb P f9895ecbf03947dc8d50afde1f9f9a1e [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:02.768396 3767 raft_consensus.cc:2454] T feff5d83990f48a0a3ddb648e03769fb P f9895ecbf03947dc8d50afde1f9f9a1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a43bf937df4caf8fa833e7f9069167 in term 1.
I20260926 06:15:02.768603 3980 leader_election.cc:302] T feff5d83990f48a0a3ddb648e03769fb P a2a43bf937df4caf8fa833e7f9069167 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a43bf937df4caf8fa833e7f9069167, f9895ecbf03947dc8d50afde1f9f9a1e; no voters:
I20260926 06:15:02.768662 3902 raft_consensus.cc:2454] T feff5d83990f48a0a3ddb648e03769fb P e758657151684f7bac40258b0e41a0c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a43bf937df4caf8fa833e7f9069167 in term 1.
I20260926 06:15:02.768738 4126 raft_consensus.cc:2781] T feff5d83990f48a0a3ddb648e03769fb P a2a43bf937df4caf8fa833e7f9069167 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:15:02.768803 4126 raft_consensus.cc:686] T feff5d83990f48a0a3ddb648e03769fb P a2a43bf937df4caf8fa833e7f9069167 [term 1 LEADER]: Becoming Leader. State: Replica: a2a43bf937df4caf8fa833e7f9069167, State: Running, Role: LEADER
I20260926 06:15:02.768860 4126 consensus_queue.cc:237] T feff5d83990f48a0a3ddb648e03769fb P a2a43bf937df4caf8fa833e7f9069167 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9895ecbf03947dc8d50afde1f9f9a1e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 36493 } } peers { permanent_uuid: "e758657151684f7bac40258b0e41a0c7" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36241 } } peers { permanent_uuid: "a2a43bf937df4caf8fa833e7f9069167" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 40247 } }
I20260926 06:15:02.769179 4123 tablet_replica.cc:439] T feff5d83990f48a0a3ddb648e03769fb P a2a43bf937df4caf8fa833e7f9069167: TxnStatusTablet state changed. Reason: New leader a2a43bf937df4caf8fa833e7f9069167. Latest consensus state: current_term: 1 leader_uuid: "a2a43bf937df4caf8fa833e7f9069167" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9895ecbf03947dc8d50afde1f9f9a1e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 36493 } } peers { permanent_uuid: "e758657151684f7bac40258b0e41a0c7" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36241 } } peers { permanent_uuid: "a2a43bf937df4caf8fa833e7f9069167" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 40247 } } }
I20260926 06:15:02.769383 4123 tablet_replica.cc:442] T feff5d83990f48a0a3ddb648e03769fb P a2a43bf937df4caf8fa833e7f9069167: This TxnStatusTablet replica's current role is: LEADER
I20260926 06:15:02.769657 4158 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 06:15:02.769812 4158 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 06:15:02.769717 3610 catalog_manager.cc:5495] T feff5d83990f48a0a3ddb648e03769fb P a2a43bf937df4caf8fa833e7f9069167 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2a43bf937df4caf8fa833e7f9069167 (127.2.110.195). New cstate: current_term: 1 leader_uuid: "a2a43bf937df4caf8fa833e7f9069167" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9895ecbf03947dc8d50afde1f9f9a1e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 36493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e758657151684f7bac40258b0e41a0c7" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a43bf937df4caf8fa833e7f9069167" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 40247 } health_report { overall_health: HEALTHY } } }
I20260926 06:15:02.790902 4113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:02.787834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54356 (local address 127.2.110.195:40247)
0926 06:15:02.787894 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:02.787897 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:02.787915 (+ 18us) server_negotiation.cc:407] Connection header received
0926 06:15:02.788007 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:02.788008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:02.788019 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:02.788075 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:15:02.788233 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.788765 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.789968 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.790321 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.790405 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:02.790611 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:15:02.790642 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:15:02.790678 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:02.790765 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 06:15:02.790797 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 06:15:02.790905 4153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:02.787718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.195:40247 (local address 127.0.0.1:54356)
0926 06:15:02.787820 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:02.787837 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:02.787906 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:02.788088 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:02.788089 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:02.788090 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 06:15:02.788155 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:02.788159 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.788847 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.788848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:02.789857 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:02.789860 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.790432 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.790434 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:02.790497 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:02.790522 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 06:15:02.790679 (+ 157us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 06:15:02.790685 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 06:15:02.790761 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 06:15:02.790816 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 06:15:02.792058 3902 raft_consensus.cc:1261] T feff5d83990f48a0a3ddb648e03769fb P e758657151684f7bac40258b0e41a0c7 [term 1 FOLLOWER]: Refusing update from remote peer a2a43bf937df4caf8fa833e7f9069167: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:15:02.792145 3767 raft_consensus.cc:1261] T feff5d83990f48a0a3ddb648e03769fb P f9895ecbf03947dc8d50afde1f9f9a1e [term 1 FOLLOWER]: Refusing update from remote peer a2a43bf937df4caf8fa833e7f9069167: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:15:02.792376 4126 consensus_queue.cc:1035] T feff5d83990f48a0a3ddb648e03769fb P a2a43bf937df4caf8fa833e7f9069167 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e758657151684f7bac40258b0e41a0c7" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:15:02.792486 4123 consensus_queue.cc:1035] T feff5d83990f48a0a3ddb648e03769fb P a2a43bf937df4caf8fa833e7f9069167 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9895ecbf03947dc8d50afde1f9f9a1e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 36493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:15:02.792936 4122 tablet_replica.cc:439] T feff5d83990f48a0a3ddb648e03769fb P e758657151684f7bac40258b0e41a0c7: TxnStatusTablet state changed. Reason: New leader a2a43bf937df4caf8fa833e7f9069167. Latest consensus state: current_term: 1 leader_uuid: "a2a43bf937df4caf8fa833e7f9069167" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9895ecbf03947dc8d50afde1f9f9a1e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 36493 } } peers { permanent_uuid: "e758657151684f7bac40258b0e41a0c7" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36241 } } peers { permanent_uuid: "a2a43bf937df4caf8fa833e7f9069167" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 40247 } } }
I20260926 06:15:02.793031 4122 tablet_replica.cc:442] T feff5d83990f48a0a3ddb648e03769fb P e758657151684f7bac40258b0e41a0c7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:15:02.793017 4124 tablet_replica.cc:439] T feff5d83990f48a0a3ddb648e03769fb P f9895ecbf03947dc8d50afde1f9f9a1e: TxnStatusTablet state changed. Reason: New leader a2a43bf937df4caf8fa833e7f9069167. Latest consensus state: current_term: 1 leader_uuid: "a2a43bf937df4caf8fa833e7f9069167" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9895ecbf03947dc8d50afde1f9f9a1e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 36493 } } peers { permanent_uuid: "e758657151684f7bac40258b0e41a0c7" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36241 } } peers { permanent_uuid: "a2a43bf937df4caf8fa833e7f9069167" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 40247 } } }
I20260926 06:15:02.793107 4124 tablet_replica.cc:442] T feff5d83990f48a0a3ddb648e03769fb P f9895ecbf03947dc8d50afde1f9f9a1e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:15:02.793748 4123 tablet_replica.cc:439] T feff5d83990f48a0a3ddb648e03769fb P a2a43bf937df4caf8fa833e7f9069167: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a2a43bf937df4caf8fa833e7f9069167" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9895ecbf03947dc8d50afde1f9f9a1e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 36493 } } peers { permanent_uuid: "e758657151684f7bac40258b0e41a0c7" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36241 } } peers { permanent_uuid: "a2a43bf937df4caf8fa833e7f9069167" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 40247 } } }
I20260926 06:15:02.793835 4123 tablet_replica.cc:442] T feff5d83990f48a0a3ddb648e03769fb P a2a43bf937df4caf8fa833e7f9069167: This TxnStatusTablet replica's current role is: LEADER
I20260926 06:15:02.794092 4124 tablet_replica.cc:439] T feff5d83990f48a0a3ddb648e03769fb P f9895ecbf03947dc8d50afde1f9f9a1e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a2a43bf937df4caf8fa833e7f9069167" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9895ecbf03947dc8d50afde1f9f9a1e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 36493 } } peers { permanent_uuid: "e758657151684f7bac40258b0e41a0c7" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36241 } } peers { permanent_uuid: "a2a43bf937df4caf8fa833e7f9069167" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 40247 } } }
I20260926 06:15:02.794173 4124 tablet_replica.cc:442] T feff5d83990f48a0a3ddb648e03769fb P f9895ecbf03947dc8d50afde1f9f9a1e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:15:02.794822 4126 tablet_replica.cc:439] T feff5d83990f48a0a3ddb648e03769fb P a2a43bf937df4caf8fa833e7f9069167: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a2a43bf937df4caf8fa833e7f9069167" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9895ecbf03947dc8d50afde1f9f9a1e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 36493 } } peers { permanent_uuid: "e758657151684f7bac40258b0e41a0c7" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36241 } } peers { permanent_uuid: "a2a43bf937df4caf8fa833e7f9069167" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 40247 } } }
I20260926 06:15:02.794920 4126 tablet_replica.cc:442] T feff5d83990f48a0a3ddb648e03769fb P a2a43bf937df4caf8fa833e7f9069167: This TxnStatusTablet replica's current role is: LEADER
I20260926 06:15:02.795120 4122 tablet_replica.cc:439] T feff5d83990f48a0a3ddb648e03769fb P e758657151684f7bac40258b0e41a0c7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a2a43bf937df4caf8fa833e7f9069167" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9895ecbf03947dc8d50afde1f9f9a1e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 36493 } } peers { permanent_uuid: "e758657151684f7bac40258b0e41a0c7" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36241 } } peers { permanent_uuid: "a2a43bf937df4caf8fa833e7f9069167" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 40247 } } }
I20260926 06:15:02.795204 4122 tablet_replica.cc:442] T feff5d83990f48a0a3ddb648e03769fb P e758657151684f7bac40258b0e41a0c7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:15:02.798298 4164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:02.795751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60738 (local address 127.2.110.194:36241)
0926 06:15:02.795917 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:02.795919 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:02.795926 (+ 7us) server_negotiation.cc:407] Connection header received
0926 06:15:02.795966 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:02.795967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:02.795983 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:02.796071 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:15:02.796181 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.796673 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.797446 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.797658 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.797742 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:02.797912 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:15:02.798031 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:15:02.798116 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:02.798156 (+ 40us) server_negotiation.cc:299] Negotiation successful
0926 06:15:02.798179 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":97,"threads_started":1}
I20260926 06:15:02.803645 4167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:02.800827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.195:40247 (local address 127.2.110.194:34251)
0926 06:15:02.801008 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:02.801017 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:02.801072 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:02.801204 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:02.801205 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:02.801206 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 06:15:02.801261 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:02.801262 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.802175 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.802176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:02.802786 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:02.802787 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.803255 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.803256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:02.803309 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:02.803330 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 06:15:02.803501 (+ 171us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 06:15:02.803502 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 06:15:02.803537 (+ 35us) client_negotiation.cc:240] Negotiation successful
0926 06:15:02.803575 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":72,"threads_started":1}
I20260926 06:15:02.803642 4113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:02.800904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:34251 (local address 127.2.110.195:40247)
0926 06:15:02.800951 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:02.800954 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:02.801084 (+ 130us) server_negotiation.cc:407] Connection header received
0926 06:15:02.801131 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:02.801132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:02.801144 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:02.801197 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:15:02.801313 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.802066 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.802871 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.803168 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.803240 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:02.803409 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:15:02.803442 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:15:02.803495 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:02.803540 (+ 45us) server_negotiation.cc:299] Negotiation successful
0926 06:15:02.803561 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 06:15:02.806123 4047 raft_consensus.cc:1261] T b5aed8266c9d4eaea0201b4ef8ccb882 P a2a43bf937df4caf8fa833e7f9069167 [term 1 FOLLOWER]: Refusing update from remote peer e758657151684f7bac40258b0e41a0c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:15:02.806483 3767 raft_consensus.cc:1261] T b5aed8266c9d4eaea0201b4ef8ccb882 P f9895ecbf03947dc8d50afde1f9f9a1e [term 1 FOLLOWER]: Refusing update from remote peer e758657151684f7bac40258b0e41a0c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:15:02.806468 4146 consensus_queue.cc:1035] T b5aed8266c9d4eaea0201b4ef8ccb882 P e758657151684f7bac40258b0e41a0c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a43bf937df4caf8fa833e7f9069167" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 40247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:15:02.806874 4146 consensus_queue.cc:1035] T b5aed8266c9d4eaea0201b4ef8ccb882 P e758657151684f7bac40258b0e41a0c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9895ecbf03947dc8d50afde1f9f9a1e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 36493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:15:02.815706 4164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:02.813190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:59625 (local address 127.2.110.194:36241)
0926 06:15:02.813233 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:02.813235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:02.813385 (+ 150us) server_negotiation.cc:407] Connection header received
0926 06:15:02.813477 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:02.813478 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:02.813489 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:02.813545 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:15:02.813676 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.814155 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.814938 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.815151 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.815239 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:02.815414 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:15:02.815452 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:15:02.815514 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:02.815592 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 06:15:02.815614 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 06:15:02.815711 4172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:02.813112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.194:36241 (local address 127.2.110.195:59625)
0926 06:15:02.813289 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:02.813300 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:02.813387 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:02.813549 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:02.813550 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:02.813551 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 06:15:02.813623 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:02.813625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.814245 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.814246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:02.814845 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:02.814847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.815239 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.815240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:02.815310 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:02.815325 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 06:15:02.815539 (+ 214us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 06:15:02.815541 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 06:15:02.815588 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 06:15:02.815635 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":88,"threads_started":1}
I20260926 06:15:02.826287 3610 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:15:02.826418 3610 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:15:02.827075 3610 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1eff30d645784947bf39eafdb0868de2: Sending DeleteTablet for 3 replicas of tablet b5aed8266c9d4eaea0201b4ef8ccb882
I20260926 06:15:02.827363 3743 tablet_service.cc:1514] Processing DeleteTablet for tablet b5aed8266c9d4eaea0201b4ef8ccb882 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:15:02 UTC) from {username='kudu', principal='kudu/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:56140
I20260926 06:15:02.827457 3887 tablet_service.cc:1514] Processing DeleteTablet for tablet b5aed8266c9d4eaea0201b4ef8ccb882 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:15:02 UTC) from {username='kudu', principal='kudu/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:60726
I20260926 06:15:02.827668 4176 tablet_replica.cc:330] T b5aed8266c9d4eaea0201b4ef8ccb882 P f9895ecbf03947dc8d50afde1f9f9a1e: stopping tablet replica
I20260926 06:15:02.827678 4027 tablet_service.cc:1514] Processing DeleteTablet for tablet b5aed8266c9d4eaea0201b4ef8ccb882 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:15:02 UTC) from {username='kudu', principal='kudu/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:54332
I20260926 06:15:02.827800 4176 raft_consensus.cc:2229] T b5aed8266c9d4eaea0201b4ef8ccb882 P f9895ecbf03947dc8d50afde1f9f9a1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:15:02.827881 4176 raft_consensus.cc:2258] T b5aed8266c9d4eaea0201b4ef8ccb882 P f9895ecbf03947dc8d50afde1f9f9a1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:15:02.828263 4176 ts_tablet_manager.cc:1905] T b5aed8266c9d4eaea0201b4ef8ccb882 P f9895ecbf03947dc8d50afde1f9f9a1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:15:02.828504 4177 tablet_replica.cc:330] T b5aed8266c9d4eaea0201b4ef8ccb882 P e758657151684f7bac40258b0e41a0c7: stopping tablet replica
I20260926 06:15:02.828581 4177 raft_consensus.cc:2229] T b5aed8266c9d4eaea0201b4ef8ccb882 P e758657151684f7bac40258b0e41a0c7 [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:15:02.828683 4177 raft_consensus.cc:2258] T b5aed8266c9d4eaea0201b4ef8ccb882 P e758657151684f7bac40258b0e41a0c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:15:02.828794 4178 tablet_replica.cc:330] T b5aed8266c9d4eaea0201b4ef8ccb882 P a2a43bf937df4caf8fa833e7f9069167: stopping tablet replica
I20260926 06:15:02.828857 4178 raft_consensus.cc:2229] T b5aed8266c9d4eaea0201b4ef8ccb882 P a2a43bf937df4caf8fa833e7f9069167 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:15:02.828902 4178 raft_consensus.cc:2258] T b5aed8266c9d4eaea0201b4ef8ccb882 P a2a43bf937df4caf8fa833e7f9069167 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:15:02.829398 4178 ts_tablet_manager.cc:1905] T b5aed8266c9d4eaea0201b4ef8ccb882 P a2a43bf937df4caf8fa833e7f9069167: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:15:02.829408 4177 ts_tablet_manager.cc:1905] T b5aed8266c9d4eaea0201b4ef8ccb882 P e758657151684f7bac40258b0e41a0c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:15:02.831221 4177 ts_tablet_manager.cc:1918] T b5aed8266c9d4eaea0201b4ef8ccb882 P e758657151684f7bac40258b0e41a0c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 06:15:02.831211 4176 ts_tablet_manager.cc:1918] T b5aed8266c9d4eaea0201b4ef8ccb882 P f9895ecbf03947dc8d50afde1f9f9a1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 06:15:02.831288 4177 log.cc:1196] T b5aed8266c9d4eaea0201b4ef8ccb882 P e758657151684f7bac40258b0e41a0c7: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/ts-1/wal/wals/b5aed8266c9d4eaea0201b4ef8ccb882
I20260926 06:15:02.831527 4177 ts_tablet_manager.cc:1939] T b5aed8266c9d4eaea0201b4ef8ccb882 P e758657151684f7bac40258b0e41a0c7: Deleting consensus metadata
I20260926 06:15:02.832284 3594 catalog_manager.cc:4841] TS e758657151684f7bac40258b0e41a0c7 (127.2.110.194:36241): tablet b5aed8266c9d4eaea0201b4ef8ccb882 (table test-table [id=05dd0089c2c247989702c11b798dfd62]) successfully deleted
I20260926 06:15:02.833464 4176 log.cc:1196] T b5aed8266c9d4eaea0201b4ef8ccb882 P f9895ecbf03947dc8d50afde1f9f9a1e: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/ts-0/wal/wals/b5aed8266c9d4eaea0201b4ef8ccb882
I20260926 06:15:02.833787 4176 ts_tablet_manager.cc:1939] T b5aed8266c9d4eaea0201b4ef8ccb882 P f9895ecbf03947dc8d50afde1f9f9a1e: Deleting consensus metadata
I20260926 06:15:02.833925 4178 ts_tablet_manager.cc:1918] T b5aed8266c9d4eaea0201b4ef8ccb882 P a2a43bf937df4caf8fa833e7f9069167: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 06:15:02.834251 3596 catalog_manager.cc:4841] TS f9895ecbf03947dc8d50afde1f9f9a1e (127.2.110.193:36493): tablet b5aed8266c9d4eaea0201b4ef8ccb882 (table test-table [id=05dd0089c2c247989702c11b798dfd62]) successfully deleted
I20260926 06:15:02.834059 4178 log.cc:1196] T b5aed8266c9d4eaea0201b4ef8ccb882 P a2a43bf937df4caf8fa833e7f9069167: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403292369990-2491-0/minicluster-data/ts-2/wal/wals/b5aed8266c9d4eaea0201b4ef8ccb882
I20260926 06:15:02.834784 4178 ts_tablet_manager.cc:1939] T b5aed8266c9d4eaea0201b4ef8ccb882 P a2a43bf937df4caf8fa833e7f9069167: Deleting consensus metadata
I20260926 06:15:02.835170 3595 catalog_manager.cc:4841] TS a2a43bf937df4caf8fa833e7f9069167 (127.2.110.195:40247): tablet b5aed8266c9d4eaea0201b4ef8ccb882 (table test-table [id=05dd0089c2c247989702c11b798dfd62]) successfully deleted
Sep 26 06:15:02 dist-test-slave-hndd krb5kdc[3565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403302, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.110.193@KRBTEST.COM
I20260926 06:15:02.844098 4115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:02.834428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56150 (local address 127.2.110.193:36493)
0926 06:15:02.834509 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:02.834511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:02.835261 (+ 750us) server_negotiation.cc:407] Connection header received
0926 06:15:02.835368 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:02.835369 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:02.835384 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:02.835433 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:02.836314 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.836839 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.837776 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.837982 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.838069 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:02.841080 (+ 3011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:02.841093 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:02.841094 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:02.841103 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:02.842878 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:02.843262 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:02.843266 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:02.843266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:02.843328 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:02.843541 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:02.843542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:02.843542 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:02.843691 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:02.843852 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:02.843897 (+ 45us) server_negotiation.cc:299] Negotiation successful
0926 06:15:02.843984 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260926 06:15:02.844491 3807 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56150
I20260926 06:15:02.852676 3660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:02.845506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51192 (local address 127.2.110.254:34967)
0926 06:15:02.845554 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:02.845556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:02.845720 (+ 164us) server_negotiation.cc:407] Connection header received
0926 06:15:02.845796 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:02.845797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:02.845808 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:02.845858 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:02.846358 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.846855 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.847544 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:02.847797 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:02.848021 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:02.849815 (+ 1794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:02.849816 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:02.849817 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:02.849827 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:02.851452 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:02.851824 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:02.851825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:02.851826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:02.851887 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:02.852105 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:02.852106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:02.852106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:02.852268 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:02.852417 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:02.852479 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 06:15:02.852564 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260926 06:15:02.853168 3610 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51192
I20260926 06:15:02.853935 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 3682
I20260926 06:15:02.859369 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 3822
I20260926 06:15:02.865269 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 3962
I20260926 06:15:02.871604 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 3578
2026-09-26T06:15:02Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3238 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:15:02 dist-test-slave-hndd krb5kdc[4185](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:15:02 dist-test-slave-hndd krb5kdc[4185](info): set up 2 sockets
Sep 26 06:15:02 dist-test-slave-hndd krb5kdc[4185](info): commencing operation
krb5kdc: starting...
W20260926 06:15:03.945075 2491 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Sep 26 06:15:04 dist-test-slave-hndd krb5kdc[4185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:15:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:15:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
I20260926 06:15:04.068782 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:43499
--webserver_interface=127.2.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:40607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.110.254:43499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:15:04.150705 4198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:04.150975 4198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:04.151014 4198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:04.152200 4198 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:15:04.152271 4198 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 06:15:04.152294 4198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:04.152324 4198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:04.152354 4198 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:15:04.152376 4198 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:15:04.153934 4198 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:40607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.110.254:43499
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:43499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:04.154196 4198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:04.154431 4198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:04.159686 4204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:04.159775 4206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:04.159826 4198 server_base.cc:1034] running on GCE node
W20260926 06:15:04.159781 4203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:04.160172 4198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:04.160532 4198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:04.161702 4198 hybrid_clock.cc:648] HybridClock initialized: now 1790403304161683 us; error 30 us; skew 500 ppm
Sep 26 06:15:04 dist-test-slave-hndd krb5kdc[4185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403304, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:04.170065 4198 init.cc:377] Logged in from keytab as kudu/127.2.110.254@KRBTEST.COM (short username kudu)
I20260926 06:15:04.170614 4198 webserver.cc:460] Webserver started at http://127.2.110.254:33167/ using document root <none> and password file <none>
I20260926 06:15:04.170887 4198 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:04.170956 4198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:04.171104 4198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:04.172230 4198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/master-0/data/instance:
uuid: "9fdc65b166ea49feab076956712a2657"
format_stamp: "Formatted at 2026-09-26 06:15:04 on dist-test-slave-hndd"
server_key: "70c8f2a0ff07349cf6ac7775d33e3c07"
server_key_iv: "fa414654ef2eb36b019bdb41d7042703"
server_key_version: "encryptionkey@0"
I20260926 06:15:04.172607 4198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/master-0/wal/instance:
uuid: "9fdc65b166ea49feab076956712a2657"
format_stamp: "Formatted at 2026-09-26 06:15:04 on dist-test-slave-hndd"
server_key: "70c8f2a0ff07349cf6ac7775d33e3c07"
server_key_iv: "fa414654ef2eb36b019bdb41d7042703"
server_key_version: "encryptionkey@0"
I20260926 06:15:04.174285 4198 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:15:04.175307 4213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:04.175516 4198 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:04.175628 4198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "9fdc65b166ea49feab076956712a2657"
format_stamp: "Formatted at 2026-09-26 06:15:04 on dist-test-slave-hndd"
server_key: "70c8f2a0ff07349cf6ac7775d33e3c07"
server_key_iv: "fa414654ef2eb36b019bdb41d7042703"
server_key_version: "encryptionkey@0"
I20260926 06:15:04.175716 4198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:04.190817 4198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:04.193269 4198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:04.193483 4198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:04.199564 4198 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.254:43499
I20260926 06:15:04.199617 4275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.254:43499 every 8 connection(s)
I20260926 06:15:04.199996 4198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:15:04.200666 4276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:04.203373 4276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fdc65b166ea49feab076956712a2657: Bootstrap starting.
I20260926 06:15:04.204043 4276 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fdc65b166ea49feab076956712a2657: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:04.204371 4276 log.cc:824] T 00000000000000000000000000000000 P 9fdc65b166ea49feab076956712a2657: Log is configured to *not* fsync() on all Append() calls
I20260926 06:15:04.204460 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 4198
I20260926 06:15:04.204592 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/master-0/wal/instance
I20260926 06:15:04.204772 2491 external_mini_cluster.cc:1428] Setting key 5ae2d88ad52d1eb6dc865d5ff914162d
I20260926 06:15:04.205143 4276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fdc65b166ea49feab076956712a2657: No bootstrap required, opened a new log
I20260926 06:15:04.207528 4276 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fdc65b166ea49feab076956712a2657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fdc65b166ea49feab076956712a2657" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 43499 } }
I20260926 06:15:04.207707 4276 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9fdc65b166ea49feab076956712a2657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:04.207787 4276 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fdc65b166ea49feab076956712a2657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fdc65b166ea49feab076956712a2657, State: Initialized, Role: FOLLOWER
I20260926 06:15:04.207959 4276 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fdc65b166ea49feab076956712a2657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fdc65b166ea49feab076956712a2657" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 43499 } }
I20260926 06:15:04.208057 4276 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9fdc65b166ea49feab076956712a2657 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:15:04.208096 4276 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9fdc65b166ea49feab076956712a2657 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:15:04.208138 4276 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fdc65b166ea49feab076956712a2657 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:04.208897 4276 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9fdc65b166ea49feab076956712a2657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fdc65b166ea49feab076956712a2657" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 43499 } }
I20260926 06:15:04.208989 4276 leader_election.cc:302] T 00000000000000000000000000000000 P 9fdc65b166ea49feab076956712a2657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9fdc65b166ea49feab076956712a2657; no voters:
I20260926 06:15:04.209203 4276 leader_election.cc:288] T 00000000000000000000000000000000 P 9fdc65b166ea49feab076956712a2657 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:15:04.209242 4281 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9fdc65b166ea49feab076956712a2657 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:15:04.209404 4281 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9fdc65b166ea49feab076956712a2657 [term 1 LEADER]: Becoming Leader. State: Replica: 9fdc65b166ea49feab076956712a2657, State: Running, Role: LEADER
I20260926 06:15:04.209513 4276 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fdc65b166ea49feab076956712a2657 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:15:04.209520 4281 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fdc65b166ea49feab076956712a2657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fdc65b166ea49feab076956712a2657" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 43499 } }
I20260926 06:15:04.209972 4283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fdc65b166ea49feab076956712a2657 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fdc65b166ea49feab076956712a2657. Latest consensus state: current_term: 1 leader_uuid: "9fdc65b166ea49feab076956712a2657" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fdc65b166ea49feab076956712a2657" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 43499 } } }
I20260926 06:15:04.210053 4283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fdc65b166ea49feab076956712a2657 [sys.catalog]: This master's current role is: LEADER
I20260926 06:15:04.210214 4282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fdc65b166ea49feab076956712a2657 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9fdc65b166ea49feab076956712a2657" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fdc65b166ea49feab076956712a2657" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 43499 } } }
I20260926 06:15:04.210285 4282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fdc65b166ea49feab076956712a2657 [sys.catalog]: This master's current role is: LEADER
I20260926 06:15:04.210407 4287 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:15:04.210922 4287 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:15:04.212627 4287 catalog_manager.cc:1347] Generated new cluster ID: eb9f840d9ec84564b64d5852b26891fe
I20260926 06:15:04.212687 4287 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 06:15:04 dist-test-slave-hndd krb5kdc[4185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:15:04.220961 4280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:04.205309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47176 (local address 127.2.110.254:43499)
0926 06:15:04.205535 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:04.205543 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:04.205570 (+ 27us) server_negotiation.cc:407] Connection header received
0926 06:15:04.207507 (+ 1937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:04.207516 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:04.207584 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:04.207784 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:04.208601 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:04.210023 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:04.211272 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:04.211958 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:04.212415 (+ 457us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:04.215974 (+ 3559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:04.215980 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:04.215980 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:04.215997 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:04.218470 (+ 2473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:04.219009 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:04.219013 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:04.219013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:04.219168 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:04.219425 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:04.219426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:04.219426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:04.219827 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:04.220449 (+ 622us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:04.220535 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 06:15:04.220715 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":132,"threads_started":1}
I20260926 06:15:04.230248 4287 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:15:04.230818 4287 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:15:04.235425 4287 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9fdc65b166ea49feab076956712a2657: Generated new TSK 0
I20260926 06:15:04.235621 4287 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab.
Entry for principal kudu/127.2.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab.
I20260926 06:15:04.272639 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.193:0
--local_ip_for_outbound_sockets=127.2.110.193
--webserver_interface=127.2.110.193
--webserver_port=0
--tserver_master_addrs=127.2.110.254:43499
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab
--principal=kudu/127.2.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:40607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:15:04.367421 4302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:04.367694 4302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:04.367741 4302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:04.369064 4302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:04.369153 4302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:04.369192 4302 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 06:15:04.369236 4302 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 06:15:04.369272 4302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.193
I20260926 06:15:04.370900 4302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:40607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab
--principal=kudu/127.2.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.110.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.110.254:43499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.193
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:04.371208 4302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:04.371460 4302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:04.377564 4308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:04.377564 4307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:04.377588 4310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:04.378023 4302 server_base.cc:1034] running on GCE node
I20260926 06:15:04.378278 4302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:04.378592 4302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:04.379781 4302 hybrid_clock.cc:648] HybridClock initialized: now 1790403304379754 us; error 39 us; skew 500 ppm
Sep 26 06:15:04 dist-test-slave-hndd krb5kdc[4185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403304, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:04.383279 4302 init.cc:377] Logged in from keytab as kudu/127.2.110.193@KRBTEST.COM (short username kudu)
I20260926 06:15:04.383847 4302 webserver.cc:460] Webserver started at http://127.2.110.193:38069/ using document root <none> and password file <none>
I20260926 06:15:04.384088 4302 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:04.384164 4302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:04.384336 4302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:04.385581 4302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-0/data/instance:
uuid: "20ab543b0b074455aae04921b83afa4e"
format_stamp: "Formatted at 2026-09-26 06:15:04 on dist-test-slave-hndd"
server_key: "165ec5820ba6f84d35383d63be20a60b"
server_key_iv: "d3b31ccf6735225382eafba2d07fd7b8"
server_key_version: "encryptionkey@0"
I20260926 06:15:04.385957 4302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-0/wal/instance:
uuid: "20ab543b0b074455aae04921b83afa4e"
format_stamp: "Formatted at 2026-09-26 06:15:04 on dist-test-slave-hndd"
server_key: "165ec5820ba6f84d35383d63be20a60b"
server_key_iv: "d3b31ccf6735225382eafba2d07fd7b8"
server_key_version: "encryptionkey@0"
I20260926 06:15:04.387413 4302 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:15:04.388715 4317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:04.389045 4302 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:04.389175 4302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-0/wal
uuid: "20ab543b0b074455aae04921b83afa4e"
format_stamp: "Formatted at 2026-09-26 06:15:04 on dist-test-slave-hndd"
server_key: "165ec5820ba6f84d35383d63be20a60b"
server_key_iv: "d3b31ccf6735225382eafba2d07fd7b8"
server_key_version: "encryptionkey@0"
I20260926 06:15:04.389261 4302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:04.398078 4302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:04.400735 4302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:04.401005 4302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:04.401862 4302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:15:04.401981 4302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:04.402100 4302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:15:04.402135 4302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 06:15:04 dist-test-slave-hndd krb5kdc[4185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403304, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.193@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:15:04.411376 4302 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.193:46297
I20260926 06:15:04.411480 4432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.193:46297 every 8 connection(s)
I20260926 06:15:04.411856 4302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
I20260926 06:15:04.412854 4280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:04.402348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:55517 (local address 127.2.110.254:43499)
0926 06:15:04.402409 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:04.402412 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:04.402751 (+ 339us) server_negotiation.cc:407] Connection header received
0926 06:15:04.403293 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:04.403295 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:04.403321 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:04.403450 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:04.404288 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:04.404816 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:04.405900 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:04.406248 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:04.406322 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:04.409162 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:04.409165 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:04.409165 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:04.409184 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:04.411101 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:04.411721 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:04.411722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:04.411722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:04.411805 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:04.412098 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:04.412099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:04.412100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:04.412272 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:04.412343 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:04.412706 (+ 363us) server_negotiation.cc:299] Negotiation successful
0926 06:15:04.412789 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 06:15:04.413106 4326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:04.402338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:43499 (local address 127.2.110.193:55517)
0926 06:15:04.402588 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:04.402633 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:04.403153 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:04.403544 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:04.403548 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:04.403887 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:04.404162 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:04.404167 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:04.404918 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:04.404919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:04.405802 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:04.405804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:04.406332 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:04.406333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:04.406470 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:04.407052 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:04.407061 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:04.409009 (+ 1948us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:04.411227 (+ 2218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:04.411233 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:04.411236 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:04.411622 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:04.411873 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:04.411874 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:04.411874 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:04.412019 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:04.412348 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:04.412358 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:04.412564 (+ 206us) client_negotiation.cc:769] Sending connection context
0926 06:15:04.412678 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 06:15:04.412827 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":121,"threads_started":1}
I20260926 06:15:04.413681 4433 heartbeater.cc:347] Connected to a master server at 127.2.110.254:43499
I20260926 06:15:04.413760 4433 heartbeater.cc:464] Registering TS with master...
I20260926 06:15:04.413970 4433 heartbeater.cc:511] Master 127.2.110.254:43499 requested a full tablet report, sending...
I20260926 06:15:04.414589 4229 ts_manager.cc:194] Registered new tserver with Master: 20ab543b0b074455aae04921b83afa4e (127.2.110.193:46297)
I20260926 06:15:04.415550 4229 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.193@KRBTEST.COM'} at 127.2.110.193:55517
I20260926 06:15:04.419229 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 4302
I20260926 06:15:04.419354 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-0/wal/instance
I20260926 06:15:04.419562 2491 external_mini_cluster.cc:1428] Setting key 3c74efa8218cd2671f121749940a8c21
I20260926 06:15:04.421367 4280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:04.414858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:60103 (local address 127.2.110.254:43499)
0926 06:15:04.414907 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:04.414909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:04.415028 (+ 119us) server_negotiation.cc:407] Connection header received
0926 06:15:04.415073 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:04.415074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:04.415084 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:04.415125 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:04.415481 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:04.416155 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:04.416970 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:04.417176 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:04.417245 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:04.418409 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:04.418411 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:04.418411 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:04.418417 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:04.420042 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:04.420464 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:04.420465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:04.420465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:04.420536 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:04.420758 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:04.420759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:04.420759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:04.420887 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:04.420969 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:04.421152 (+ 183us) server_negotiation.cc:299] Negotiation successful
0926 06:15:04.421240 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 06:15:04.421376 4439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:04.414795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:43499 (local address 127.2.110.193:60103)
0926 06:15:04.414948 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:04.414959 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:04.415025 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:04.415141 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:04.415141 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:04.415344 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:04.415409 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:04.415411 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:04.416268 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:04.416269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:04.416880 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:04.416883 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:04.417278 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:04.417279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:04.417414 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:04.417964 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:04.417970 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:04.418313 (+ 343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:04.420113 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:04.420115 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:04.420115 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:04.420359 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:04.420610 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:04.420611 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:04.420611 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:04.420688 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:04.420959 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:04.420960 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:04.421017 (+ 57us) client_negotiation.cc:769] Sending connection context
0926 06:15:04.421103 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 06:15:04.421233 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":100,"threads_started":1}
WARNING: no policy specified for kudu/127.2.110.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab.
Entry for principal kudu/127.2.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab.
I20260926 06:15:04.448352 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.194:0
--local_ip_for_outbound_sockets=127.2.110.194
--webserver_interface=127.2.110.194
--webserver_port=0
--tserver_master_addrs=127.2.110.254:43499
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab
--principal=kudu/127.2.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:40607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:15:04.535671 4442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:04.535943 4442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:04.535992 4442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:04.537243 4442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:04.537319 4442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:04.537354 4442 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 06:15:04.537439 4442 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 06:15:04.537480 4442 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.194
I20260926 06:15:04.539033 4442 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:40607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab
--principal=kudu/127.2.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.110.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.110.254:43499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.194
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:04.539326 4442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:04.539563 4442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:04.545686 4448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:04.545710 4447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:04.545710 4450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:04.546402 4442 server_base.cc:1034] running on GCE node
I20260926 06:15:04.546634 4442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:04.546907 4442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:04.548069 4442 hybrid_clock.cc:648] HybridClock initialized: now 1790403304548054 us; error 30 us; skew 500 ppm
Sep 26 06:15:04 dist-test-slave-hndd krb5kdc[4185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403304, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:04.550885 4442 init.cc:377] Logged in from keytab as kudu/127.2.110.194@KRBTEST.COM (short username kudu)
I20260926 06:15:04.551375 4442 webserver.cc:460] Webserver started at http://127.2.110.194:46029/ using document root <none> and password file <none>
I20260926 06:15:04.551595 4442 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:04.551640 4442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:04.551810 4442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:04.553283 4442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-1/data/instance:
uuid: "19dab98ec7184b578b218bb35dcccc85"
format_stamp: "Formatted at 2026-09-26 06:15:04 on dist-test-slave-hndd"
server_key: "6062af5d0ce5f4a2540902e447b6de67"
server_key_iv: "7b8cf02530871090bbc5fe46641508bf"
server_key_version: "encryptionkey@0"
I20260926 06:15:04.553712 4442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-1/wal/instance:
uuid: "19dab98ec7184b578b218bb35dcccc85"
format_stamp: "Formatted at 2026-09-26 06:15:04 on dist-test-slave-hndd"
server_key: "6062af5d0ce5f4a2540902e447b6de67"
server_key_iv: "7b8cf02530871090bbc5fe46641508bf"
server_key_version: "encryptionkey@0"
I20260926 06:15:04.555076 4442 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:04.555883 4457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:04.556143 4442 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:04.556248 4442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-1/wal
uuid: "19dab98ec7184b578b218bb35dcccc85"
format_stamp: "Formatted at 2026-09-26 06:15:04 on dist-test-slave-hndd"
server_key: "6062af5d0ce5f4a2540902e447b6de67"
server_key_iv: "7b8cf02530871090bbc5fe46641508bf"
server_key_version: "encryptionkey@0"
I20260926 06:15:04.556330 4442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:04.576066 4442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:04.578043 4442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:04.578230 4442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:04.578811 4442 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:15:04.578873 4442 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:04.578918 4442 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:15:04.578950 4442 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 06:15:04 dist-test-slave-hndd krb5kdc[4185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403304, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.194@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:15:04.586052 4442 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.194:45967
I20260926 06:15:04.586452 4572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.194:45967 every 8 connection(s)
I20260926 06:15:04.586515 4442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
I20260926 06:15:04.588399 4280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:04.579216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:58571 (local address 127.2.110.254:43499)
0926 06:15:04.579280 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:04.579282 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:04.579502 (+ 220us) server_negotiation.cc:407] Connection header received
0926 06:15:04.580078 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:04.580079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:04.580092 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:04.580149 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:04.581033 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:04.581919 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:04.582927 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:04.583115 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:04.583200 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:04.585610 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:04.585612 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:04.585613 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:04.585620 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:04.587250 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:04.587607 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:04.587607 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:04.587608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:04.587660 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:04.587842 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:04.587842 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:04.587843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:04.587946 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:04.587997 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:04.588263 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 06:15:04.588335 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 06:15:04.588471 4466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:04.579219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:43499 (local address 127.2.110.194:58571)
0926 06:15:04.579382 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:04.579425 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:04.579978 (+ 553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:04.580228 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:04.580232 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:04.580616 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:04.580917 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:04.580923 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:04.582019 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:04.582020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:04.582821 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:04.582823 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:04.583244 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:04.583245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:04.583413 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:04.583979 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:04.583987 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:04.585519 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:04.587327 (+ 1808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:04.587332 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:04.587333 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:04.587542 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:04.587705 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:04.587705 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:04.587705 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:04.587795 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:04.588002 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:04.588007 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:04.588115 (+ 108us) client_negotiation.cc:769] Sending connection context
0926 06:15:04.588197 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 06:15:04.588319 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":82,"threads_started":1}
I20260926 06:15:04.588963 4573 heartbeater.cc:347] Connected to a master server at 127.2.110.254:43499
I20260926 06:15:04.589044 4573 heartbeater.cc:464] Registering TS with master...
I20260926 06:15:04.589234 4573 heartbeater.cc:511] Master 127.2.110.254:43499 requested a full tablet report, sending...
I20260926 06:15:04.589632 4230 ts_manager.cc:194] Registered new tserver with Master: 19dab98ec7184b578b218bb35dcccc85 (127.2.110.194:45967)
I20260926 06:15:04.590150 4230 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.194@KRBTEST.COM'} at 127.2.110.194:58571
I20260926 06:15:04.593655 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 4442
I20260926 06:15:04.593773 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-1/wal/instance
I20260926 06:15:04.593984 2491 external_mini_cluster.cc:1428] Setting key 4a48857726cfde887e2328ce6d9cf44d
I20260926 06:15:04.596194 4280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:04.590088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:44259 (local address 127.2.110.254:43499)
0926 06:15:04.590132 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:04.590134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:04.590237 (+ 103us) server_negotiation.cc:407] Connection header received
0926 06:15:04.590305 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:04.590305 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:04.590317 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:04.590384 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:04.590754 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:04.591223 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:04.591869 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:04.592043 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:04.592092 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:04.593068 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:04.593069 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:04.593069 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:04.593074 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:04.594638 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:04.594943 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:04.594945 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:04.594945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:04.595008 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:04.595394 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:04.595395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:04.595396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:04.595604 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:04.595689 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:04.596028 (+ 339us) server_negotiation.cc:299] Negotiation successful
0926 06:15:04.596115 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 06:15:04.596320 4579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:04.590033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:43499 (local address 127.2.110.194:44259)
0926 06:15:04.590162 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:04.590174 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:04.590248 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:04.590395 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:04.590395 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:04.590606 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:04.590676 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:04.590678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:04.591307 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:04.591308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:04.591803 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:04.591805 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:04.592105 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:04.592106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:04.592166 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:04.592683 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:04.592689 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:04.592997 (+ 308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:04.594713 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:04.594714 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:04.594714 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:04.594873 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:04.595158 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:04.595159 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:04.595160 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:04.595281 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:04.595710 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:04.595712 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:04.595893 (+ 181us) client_negotiation.cc:769] Sending connection context
0926 06:15:04.595989 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 06:15:04.596131 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":75,"threads_started":1}
WARNING: no policy specified for kudu/127.2.110.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab.
Entry for principal kudu/127.2.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab.
I20260926 06:15:04.620591 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.195:0
--local_ip_for_outbound_sockets=127.2.110.195
--webserver_interface=127.2.110.195
--webserver_port=0
--tserver_master_addrs=127.2.110.254:43499
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab
--principal=kudu/127.2.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:40607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:15:04.705826 4582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:04.706049 4582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:04.706079 4582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:04.707319 4582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:04.707367 4582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:04.707391 4582 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 06:15:04.707425 4582 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 06:15:04.707461 4582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.195
I20260926 06:15:04.709005 4582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:40607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab
--principal=kudu/127.2.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.110.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.110.254:43499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.195
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:04.709393 4582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:04.709646 4582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:04.715276 4587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:04.715405 4590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:04.715337 4588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:04.715531 4582 server_base.cc:1034] running on GCE node
I20260926 06:15:04.715804 4582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:04.716050 4582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:04.717221 4582 hybrid_clock.cc:648] HybridClock initialized: now 1790403304717199 us; error 28 us; skew 500 ppm
Sep 26 06:15:04 dist-test-slave-hndd krb5kdc[4185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403304, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:04.720431 4582 init.cc:377] Logged in from keytab as kudu/127.2.110.195@KRBTEST.COM (short username kudu)
I20260926 06:15:04.720974 4582 webserver.cc:460] Webserver started at http://127.2.110.195:34471/ using document root <none> and password file <none>
I20260926 06:15:04.721233 4582 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:04.721303 4582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:04.721498 4582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:04.722575 4582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-2/data/instance:
uuid: "9736545427474bbbabc55a4fcfb264de"
format_stamp: "Formatted at 2026-09-26 06:15:04 on dist-test-slave-hndd"
server_key: "6afc2674e6c91ed58cfd9b177904221d"
server_key_iv: "f212d5b2b0434bda26265d409ea4f703"
server_key_version: "encryptionkey@0"
I20260926 06:15:04.722939 4582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-2/wal/instance:
uuid: "9736545427474bbbabc55a4fcfb264de"
format_stamp: "Formatted at 2026-09-26 06:15:04 on dist-test-slave-hndd"
server_key: "6afc2674e6c91ed58cfd9b177904221d"
server_key_iv: "f212d5b2b0434bda26265d409ea4f703"
server_key_version: "encryptionkey@0"
I20260926 06:15:04.724339 4582 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:15:04.725229 4597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:04.725577 4582 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:04.725667 4582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-2/wal
uuid: "9736545427474bbbabc55a4fcfb264de"
format_stamp: "Formatted at 2026-09-26 06:15:04 on dist-test-slave-hndd"
server_key: "6afc2674e6c91ed58cfd9b177904221d"
server_key_iv: "f212d5b2b0434bda26265d409ea4f703"
server_key_version: "encryptionkey@0"
I20260926 06:15:04.725776 4582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:04.737079 4582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:04.739127 4582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:04.739290 4582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:04.739804 4582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:15:04.739868 4582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:04.739913 4582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:15:04.739938 4582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:04.746975 4582 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.195:36447
I20260926 06:15:04.747246 4712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.195:36447 every 8 connection(s)
Sep 26 06:15:04 dist-test-slave-hndd krb5kdc[4185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403304, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.195@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:15:04.747716 4582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
I20260926 06:15:04.750767 4280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:04.740291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:56107 (local address 127.2.110.254:43499)
0926 06:15:04.740345 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:04.740347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:04.740570 (+ 223us) server_negotiation.cc:407] Connection header received
0926 06:15:04.741114 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:04.741115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:04.741129 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:04.741180 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:04.742099 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:04.742987 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:04.744107 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:04.744450 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:04.744524 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:04.748036 (+ 3512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:04.748038 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:04.748038 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:04.748048 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:04.749327 (+ 1279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:04.749786 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:04.749787 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:04.749788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:04.749847 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:04.750100 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:04.750101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:04.750101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:04.750222 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:04.750290 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:04.750576 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 06:15:04.750664 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 06:15:04.750948 4606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:04.740290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:43499 (local address 127.2.110.195:56107)
0926 06:15:04.740453 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:04.740488 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:04.741006 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:04.741251 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:04.741256 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:04.741682 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:04.741988 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:04.741995 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:04.743091 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:04.743092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:04.743979 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:04.743981 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:04.744574 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:04.744575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:04.744760 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:04.745959 (+ 1199us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:04.745971 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:04.747919 (+ 1948us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:04.749459 (+ 1540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:04.749466 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:04.749468 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:04.749694 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:04.749912 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:04.749913 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:04.749913 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:04.750026 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:04.750292 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:04.750298 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:04.750439 (+ 141us) client_negotiation.cc:769] Sending connection context
0926 06:15:04.750541 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 06:15:04.750707 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":95,"threads_started":1}
I20260926 06:15:04.751469 4713 heartbeater.cc:347] Connected to a master server at 127.2.110.254:43499
I20260926 06:15:04.751550 4713 heartbeater.cc:464] Registering TS with master...
I20260926 06:15:04.751747 4713 heartbeater.cc:511] Master 127.2.110.254:43499 requested a full tablet report, sending...
I20260926 06:15:04.752190 4229 ts_manager.cc:194] Registered new tserver with Master: 9736545427474bbbabc55a4fcfb264de (127.2.110.195:36447)
I20260926 06:15:04.752583 4229 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.195@KRBTEST.COM'} at 127.2.110.195:56107
I20260926 06:15:04.756052 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 4582
I20260926 06:15:04.756148 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-2/wal/instance
I20260926 06:15:04.756363 2491 external_mini_cluster.cc:1428] Setting key 40d60c5ecce334ffa6d7b13d532e0837
I20260926 06:15:04.757263 2491 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 06:15:04.758702 4280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:04.752627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:36591 (local address 127.2.110.254:43499)
0926 06:15:04.752670 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:04.752672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:04.752799 (+ 127us) server_negotiation.cc:407] Connection header received
0926 06:15:04.752871 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:04.752872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:04.752883 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:04.752929 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:04.753247 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:04.753804 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:04.754577 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:04.754784 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:04.754866 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:04.755809 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:04.755810 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:04.755811 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:04.755818 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:04.757564 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:04.757912 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:04.757914 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:04.757914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:04.757971 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:04.758166 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:04.758167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:04.758168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:04.758301 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:04.758370 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:04.758526 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 06:15:04.758608 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 06:15:04.758719 4719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:04.752578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:43499 (local address 127.2.110.195:36591)
0926 06:15:04.752735 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:04.752745 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:04.752814 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:04.752941 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:04.752942 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:04.753118 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:04.753172 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:04.753173 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:04.753898 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:04.753899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:04.754475 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:04.754477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:04.754868 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:04.754869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:04.754944 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:04.755420 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:04.755425 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:04.755721 (+ 296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:04.757646 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:04.757647 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:04.757647 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:04.757828 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:04.758040 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:04.758041 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:04.758041 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:04.758100 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:04.758373 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:04.758374 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:04.758423 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 06:15:04.758501 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 06:15:04.758610 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":65,"threads_started":1}
Sep 26 06:15:04 dist-test-slave-hndd krb5kdc[4185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 06:15:04 dist-test-slave-hndd krb5kdc[4185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:15:04.784163 4280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:04.775510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47178 (local address 127.2.110.254:43499)
0926 06:15:04.775574 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:04.775576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:04.775590 (+ 14us) server_negotiation.cc:407] Connection header received
0926 06:15:04.775654 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:04.775655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:04.775671 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:04.775722 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:04.776374 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:04.776870 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:04.777961 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:04.778202 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:04.778313 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:04.781237 (+ 2924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:04.781239 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:04.781239 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:04.781262 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:04.782805 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:04.783286 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:04.783288 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:04.783289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:04.783361 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:04.783539 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:04.783540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:04.783540 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:04.783665 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:04.783766 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:04.783995 (+ 229us) server_negotiation.cc:299] Negotiation successful
0926 06:15:04.784091 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 06:15:04.785986 4229 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47178:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:15:04.787735 4229 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:15:04.794801 4734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:04.790516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59584 (local address 127.2.110.195:36447)
0926 06:15:04.790682 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:04.790686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:04.791000 (+ 314us) server_negotiation.cc:407] Connection header received
0926 06:15:04.791042 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:04.791044 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:04.791128 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:04.791250 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:15:04.791434 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:04.792525 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:04.793803 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:04.794329 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:04.794463 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:04.794529 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:04.794659 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 06:15:04.794703 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":92,"threads_started":1}
I20260926 06:15:04.794776 4732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:04.790284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.195:36447 (local address 127.0.0.1:59584)
0926 06:15:04.790881 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:04.790888 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:04.790946 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:04.791315 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:04.791316 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:04.791319 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:15:04.791354 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:04.791355 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:04.792592 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:04.792594 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:04.793722 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:04.793725 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:04.794469 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:04.794470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:04.794547 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:04.794562 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 06:15:04.794627 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 06:15:04.794683 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":102,"threads_started":1}
I20260926 06:15:04.795154 4735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:04.790530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35210 (local address 127.2.110.193:46297)
0926 06:15:04.790703 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:04.790708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:04.790840 (+ 132us) server_negotiation.cc:407] Connection header received
0926 06:15:04.790882 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:04.790884 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:04.790900 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:04.790968 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:15:04.791233 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:04.792449 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:04.793850 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:04.794598 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:04.794670 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:04.794730 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:04.794999 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 06:15:04.795046 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":79,"threads_started":1}
I20260926 06:15:04.795300 4733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:04.790373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32984 (local address 127.2.110.194:45967)
0926 06:15:04.790533 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:04.790539 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:04.790770 (+ 231us) server_negotiation.cc:407] Connection header received
0926 06:15:04.790825 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:04.790828 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:04.790845 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:04.790928 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:15:04.791254 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:04.792090 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:04.793849 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:04.794598 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:04.794670 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:04.794722 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:04.795138 (+ 416us) server_negotiation.cc:299] Negotiation successful
0926 06:15:04.795184 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":92,"threads_started":1}
I20260926 06:15:04.795058 4730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:04.790260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.194:45967 (local address 127.0.0.1:32984)
0926 06:15:04.790558 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:04.790596 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:04.790672 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:04.791028 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:04.791033 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:04.791036 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:15:04.791157 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:04.791161 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:04.792174 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:04.792176 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:04.793742 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:04.793745 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:04.794759 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:04.794761 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:04.794847 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:04.794849 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 06:15:04.794896 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 06:15:04.794952 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":48,"thread_start_us":58,"threads_started":1}
I20260926 06:15:04.795073 4731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:04.790280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.193:46297 (local address 127.0.0.1:35210)
0926 06:15:04.790756 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:04.790771 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:04.790825 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:04.791030 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:04.791036 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:04.791038 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:15:04.791138 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:04.791140 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:04.792529 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:04.792530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:04.793743 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:04.793745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:04.794762 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:04.794762 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:04.794839 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:04.794843 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:15:04.794896 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 06:15:04.794952 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"mutex_wait_us":32,"thread_start_us":101,"threads_started":1}
I20260926 06:15:04.795944 4647 tablet_service.cc:1467] Processing CreateTablet for tablet 165fd098bae848da9929ae1827f03962 (DEFAULT_TABLE table=test-table [id=7f74433c00134b2286e5edb2f3f003a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:15:04.796049 4507 tablet_service.cc:1467] Processing CreateTablet for tablet 165fd098bae848da9929ae1827f03962 (DEFAULT_TABLE table=test-table [id=7f74433c00134b2286e5edb2f3f003a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:15:04.796031 4367 tablet_service.cc:1467] Processing CreateTablet for tablet 165fd098bae848da9929ae1827f03962 (DEFAULT_TABLE table=test-table [id=7f74433c00134b2286e5edb2f3f003a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:15:04.796317 4647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 165fd098bae848da9929ae1827f03962. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:04.796327 4507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 165fd098bae848da9929ae1827f03962. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:04.796401 4367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 165fd098bae848da9929ae1827f03962. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:04.799448 4736 tablet_bootstrap.cc:492] T 165fd098bae848da9929ae1827f03962 P 20ab543b0b074455aae04921b83afa4e: Bootstrap starting.
I20260926 06:15:04.799857 4737 tablet_bootstrap.cc:492] T 165fd098bae848da9929ae1827f03962 P 9736545427474bbbabc55a4fcfb264de: Bootstrap starting.
I20260926 06:15:04.799993 4738 tablet_bootstrap.cc:492] T 165fd098bae848da9929ae1827f03962 P 19dab98ec7184b578b218bb35dcccc85: Bootstrap starting.
I20260926 06:15:04.800443 4736 tablet_bootstrap.cc:654] T 165fd098bae848da9929ae1827f03962 P 20ab543b0b074455aae04921b83afa4e: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:04.800595 4738 tablet_bootstrap.cc:654] T 165fd098bae848da9929ae1827f03962 P 19dab98ec7184b578b218bb35dcccc85: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:04.800678 4737 tablet_bootstrap.cc:654] T 165fd098bae848da9929ae1827f03962 P 9736545427474bbbabc55a4fcfb264de: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:04.800808 4736 log.cc:824] T 165fd098bae848da9929ae1827f03962 P 20ab543b0b074455aae04921b83afa4e: Log is configured to *not* fsync() on all Append() calls
I20260926 06:15:04.800906 4738 log.cc:824] T 165fd098bae848da9929ae1827f03962 P 19dab98ec7184b578b218bb35dcccc85: Log is configured to *not* fsync() on all Append() calls
I20260926 06:15:04.801023 4737 log.cc:824] T 165fd098bae848da9929ae1827f03962 P 9736545427474bbbabc55a4fcfb264de: Log is configured to *not* fsync() on all Append() calls
I20260926 06:15:04.801645 4738 tablet_bootstrap.cc:492] T 165fd098bae848da9929ae1827f03962 P 19dab98ec7184b578b218bb35dcccc85: No bootstrap required, opened a new log
I20260926 06:15:04.801680 4736 tablet_bootstrap.cc:492] T 165fd098bae848da9929ae1827f03962 P 20ab543b0b074455aae04921b83afa4e: No bootstrap required, opened a new log
I20260926 06:15:04.801710 4737 tablet_bootstrap.cc:492] T 165fd098bae848da9929ae1827f03962 P 9736545427474bbbabc55a4fcfb264de: No bootstrap required, opened a new log
I20260926 06:15:04.801739 4738 ts_tablet_manager.cc:1397] T 165fd098bae848da9929ae1827f03962 P 19dab98ec7184b578b218bb35dcccc85: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 06:15:04.801756 4736 ts_tablet_manager.cc:1397] T 165fd098bae848da9929ae1827f03962 P 20ab543b0b074455aae04921b83afa4e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 06:15:04.801767 4737 ts_tablet_manager.cc:1397] T 165fd098bae848da9929ae1827f03962 P 9736545427474bbbabc55a4fcfb264de: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 06:15:04.803192 4738 raft_consensus.cc:348] T 165fd098bae848da9929ae1827f03962 P 19dab98ec7184b578b218bb35dcccc85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } } peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } }
I20260926 06:15:04.803356 4738 raft_consensus.cc:374] T 165fd098bae848da9929ae1827f03962 P 19dab98ec7184b578b218bb35dcccc85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:04.803400 4738 raft_consensus.cc:729] T 165fd098bae848da9929ae1827f03962 P 19dab98ec7184b578b218bb35dcccc85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19dab98ec7184b578b218bb35dcccc85, State: Initialized, Role: FOLLOWER
I20260926 06:15:04.803510 4738 consensus_queue.cc:260] T 165fd098bae848da9929ae1827f03962 P 19dab98ec7184b578b218bb35dcccc85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } } peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } }
I20260926 06:15:04.803803 4738 ts_tablet_manager.cc:1428] T 165fd098bae848da9929ae1827f03962 P 19dab98ec7184b578b218bb35dcccc85: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:15:04.803745 4737 raft_consensus.cc:348] T 165fd098bae848da9929ae1827f03962 P 9736545427474bbbabc55a4fcfb264de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } } peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } }
I20260926 06:15:04.803750 4736 raft_consensus.cc:348] T 165fd098bae848da9929ae1827f03962 P 20ab543b0b074455aae04921b83afa4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } } peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } }
I20260926 06:15:04.803900 4736 raft_consensus.cc:374] T 165fd098bae848da9929ae1827f03962 P 20ab543b0b074455aae04921b83afa4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:04.803900 4737 raft_consensus.cc:374] T 165fd098bae848da9929ae1827f03962 P 9736545427474bbbabc55a4fcfb264de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:04.803946 4736 raft_consensus.cc:729] T 165fd098bae848da9929ae1827f03962 P 20ab543b0b074455aae04921b83afa4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20ab543b0b074455aae04921b83afa4e, State: Initialized, Role: FOLLOWER
I20260926 06:15:04.803946 4737 raft_consensus.cc:729] T 165fd098bae848da9929ae1827f03962 P 9736545427474bbbabc55a4fcfb264de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9736545427474bbbabc55a4fcfb264de, State: Initialized, Role: FOLLOWER
I20260926 06:15:04.804059 4736 consensus_queue.cc:260] T 165fd098bae848da9929ae1827f03962 P 20ab543b0b074455aae04921b83afa4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } } peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } }
I20260926 06:15:04.804059 4737 consensus_queue.cc:260] T 165fd098bae848da9929ae1827f03962 P 9736545427474bbbabc55a4fcfb264de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } } peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } }
I20260926 06:15:04.804327 4736 ts_tablet_manager.cc:1428] T 165fd098bae848da9929ae1827f03962 P 20ab543b0b074455aae04921b83afa4e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 06:15:04.804327 4737 ts_tablet_manager.cc:1428] T 165fd098bae848da9929ae1827f03962 P 9736545427474bbbabc55a4fcfb264de: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 06:15:04.804373 4433 heartbeater.cc:503] Master 127.2.110.254:43499 was elected leader, sending a full tablet report...
I20260926 06:15:04.804440 4713 heartbeater.cc:503] Master 127.2.110.254:43499 was elected leader, sending a full tablet report...
I20260926 06:15:04.804742 4573 heartbeater.cc:503] Master 127.2.110.254:43499 was elected leader, sending a full tablet report...
W20260926 06:15:04.837134 4574 tablet.cc:2369] T 165fd098bae848da9929ae1827f03962 P 19dab98ec7184b578b218bb35dcccc85: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 06:15:04.912648 4434 tablet.cc:2369] T 165fd098bae848da9929ae1827f03962 P 20ab543b0b074455aae04921b83afa4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:15:04.919124 4744 raft_consensus.cc:482] T 165fd098bae848da9929ae1827f03962 P 9736545427474bbbabc55a4fcfb264de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:15:04.919245 4744 raft_consensus.cc:504] T 165fd098bae848da9929ae1827f03962 P 9736545427474bbbabc55a4fcfb264de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } } peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } }
I20260926 06:15:04.919739 4744 leader_election.cc:288] T 165fd098bae848da9929ae1827f03962 P 9736545427474bbbabc55a4fcfb264de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19dab98ec7184b578b218bb35dcccc85 (127.2.110.194:45967), 20ab543b0b074455aae04921b83afa4e (127.2.110.193:46297)
I20260926 06:15:04.923987 4733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:04.919994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:53267 (local address 127.2.110.194:45967)
0926 06:15:04.920045 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:04.920047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:04.920060 (+ 13us) server_negotiation.cc:407] Connection header received
0926 06:15:04.920133 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:04.920134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:04.920148 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:04.920206 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:15:04.920389 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:04.921153 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:04.922798 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:04.923567 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:04.923668 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:04.923677 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:04.923868 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 06:15:04.923915 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 06:15:04.923990 4735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:04.920022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:34943 (local address 127.2.110.193:46297)
0926 06:15:04.920065 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:04.920067 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:04.920246 (+ 179us) server_negotiation.cc:407] Connection header received
0926 06:15:04.920278 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:04.920279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:04.920289 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:04.920338 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:15:04.920487 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:04.921176 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:04.922798 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:04.923564 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:04.923660 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:04.923671 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:04.923889 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 06:15:04.923931 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 06:15:04.923990 4606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:04.919938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.194:45967 (local address 127.2.110.195:53267)
0926 06:15:04.919997 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:04.920009 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:04.920054 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:04.920232 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:04.920233 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:04.920235 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:15:04.920308 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:04.920309 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:04.921301 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:04.921302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:04.922610 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:04.922613 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:04.923723 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:04.923724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:04.923794 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:04.923797 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:15:04.923840 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 06:15:04.923906 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":38}
I20260926 06:15:04.924005 4745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:04.919972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.193:46297 (local address 127.2.110.195:34943)
0926 06:15:04.920148 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:04.920157 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:04.920231 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:04.920356 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:04.920356 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:04.920358 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:15:04.920419 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:04.920420 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:04.921283 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:04.921285 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:04.922610 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:04.922613 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:04.923678 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:04.923679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:04.923783 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:04.923788 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 06:15:04.923872 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 06:15:04.923914 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":86,"thread_start_us":115,"threads_started":1}
I20260926 06:15:04.924371 4387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "165fd098bae848da9929ae1827f03962" candidate_uuid: "9736545427474bbbabc55a4fcfb264de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20ab543b0b074455aae04921b83afa4e" is_pre_election: true
I20260926 06:15:04.924372 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "165fd098bae848da9929ae1827f03962" candidate_uuid: "9736545427474bbbabc55a4fcfb264de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19dab98ec7184b578b218bb35dcccc85" is_pre_election: true
I20260926 06:15:04.924546 4387 raft_consensus.cc:2454] T 165fd098bae848da9929ae1827f03962 P 20ab543b0b074455aae04921b83afa4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9736545427474bbbabc55a4fcfb264de in term 0.
I20260926 06:15:04.924546 4527 raft_consensus.cc:2454] T 165fd098bae848da9929ae1827f03962 P 19dab98ec7184b578b218bb35dcccc85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9736545427474bbbabc55a4fcfb264de in term 0.
I20260926 06:15:04.924793 4600 leader_election.cc:302] T 165fd098bae848da9929ae1827f03962 P 9736545427474bbbabc55a4fcfb264de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20ab543b0b074455aae04921b83afa4e, 9736545427474bbbabc55a4fcfb264de; no voters:
I20260926 06:15:04.924933 4744 raft_consensus.cc:2781] T 165fd098bae848da9929ae1827f03962 P 9736545427474bbbabc55a4fcfb264de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:15:04.925007 4744 raft_consensus.cc:482] T 165fd098bae848da9929ae1827f03962 P 9736545427474bbbabc55a4fcfb264de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:15:04.925150 4744 raft_consensus.cc:3037] T 165fd098bae848da9929ae1827f03962 P 9736545427474bbbabc55a4fcfb264de [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:04.925935 4744 raft_consensus.cc:504] T 165fd098bae848da9929ae1827f03962 P 9736545427474bbbabc55a4fcfb264de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } } peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } }
I20260926 06:15:04.926249 4744 leader_election.cc:288] T 165fd098bae848da9929ae1827f03962 P 9736545427474bbbabc55a4fcfb264de [CANDIDATE]: Term 1 election: Requested vote from peers 19dab98ec7184b578b218bb35dcccc85 (127.2.110.194:45967), 20ab543b0b074455aae04921b83afa4e (127.2.110.193:46297)
I20260926 06:15:04.926358 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "165fd098bae848da9929ae1827f03962" candidate_uuid: "9736545427474bbbabc55a4fcfb264de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19dab98ec7184b578b218bb35dcccc85"
I20260926 06:15:04.926436 4387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "165fd098bae848da9929ae1827f03962" candidate_uuid: "9736545427474bbbabc55a4fcfb264de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20ab543b0b074455aae04921b83afa4e"
I20260926 06:15:04.926445 4527 raft_consensus.cc:3037] T 165fd098bae848da9929ae1827f03962 P 19dab98ec7184b578b218bb35dcccc85 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:04.926503 4387 raft_consensus.cc:3037] T 165fd098bae848da9929ae1827f03962 P 20ab543b0b074455aae04921b83afa4e [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:04.927336 4527 raft_consensus.cc:2454] T 165fd098bae848da9929ae1827f03962 P 19dab98ec7184b578b218bb35dcccc85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9736545427474bbbabc55a4fcfb264de in term 1.
I20260926 06:15:04.927335 4387 raft_consensus.cc:2454] T 165fd098bae848da9929ae1827f03962 P 20ab543b0b074455aae04921b83afa4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9736545427474bbbabc55a4fcfb264de in term 1.
I20260926 06:15:04.927528 4600 leader_election.cc:302] T 165fd098bae848da9929ae1827f03962 P 9736545427474bbbabc55a4fcfb264de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20ab543b0b074455aae04921b83afa4e, 9736545427474bbbabc55a4fcfb264de; no voters:
I20260926 06:15:04.927634 4744 raft_consensus.cc:2781] T 165fd098bae848da9929ae1827f03962 P 9736545427474bbbabc55a4fcfb264de [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:15:04.927779 4744 raft_consensus.cc:686] T 165fd098bae848da9929ae1827f03962 P 9736545427474bbbabc55a4fcfb264de [term 1 LEADER]: Becoming Leader. State: Replica: 9736545427474bbbabc55a4fcfb264de, State: Running, Role: LEADER
I20260926 06:15:04.927884 4744 consensus_queue.cc:237] T 165fd098bae848da9929ae1827f03962 P 9736545427474bbbabc55a4fcfb264de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } } peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } }
I20260926 06:15:04.928785 4228 catalog_manager.cc:5495] T 165fd098bae848da9929ae1827f03962 P 9736545427474bbbabc55a4fcfb264de reported cstate change: term changed from 0 to 1, leader changed from <none> to 9736545427474bbbabc55a4fcfb264de (127.2.110.195). New cstate: current_term: 1 leader_uuid: "9736545427474bbbabc55a4fcfb264de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } health_report { overall_health: HEALTHY } } }
Sep 26 06:15:04 dist-test-slave-hndd krb5kdc[4185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403304, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.254@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:15:04.985643 4280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:04.978375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47184 (local address 127.2.110.254:43499)
0926 06:15:04.978463 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:04.978466 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:04.978577 (+ 111us) server_negotiation.cc:407] Connection header received
0926 06:15:04.978673 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:04.978674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:04.978684 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:04.978765 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:04.979273 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:04.979816 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:04.980510 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:04.980692 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:04.980771 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:04.983033 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:04.983034 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:04.983034 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:04.983041 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:04.984417 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:04.984832 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:04.984833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:04.984833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:04.984888 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:04.985072 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:04.985073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:04.985073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:04.985187 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:04.985259 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:04.985443 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 06:15:04.985526 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 06:15:04.985721 4753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:04.978274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:43499 (local address 127.0.0.1:47184)
0926 06:15:04.978493 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:04.978505 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:04.978599 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:04.978772 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:04.978772 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:04.979124 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:04.979194 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:04.979201 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:04.979899 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:04.979900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:04.980430 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:04.980432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:04.980770 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:04.980771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:04.980821 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:04.981396 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:04.981401 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:04.982928 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:04.984539 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:04.984547 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:04.984549 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:04.984760 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:04.984957 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:04.984957 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:04.984958 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:04.985016 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:04.985249 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:04.985255 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:04.985300 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 06:15:04.985419 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 06:15:04.985585 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":63,"thread_start_us":103,"threads_started":1}
I20260926 06:15:04.987928 4228 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:47184:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 06:15:04.988193 4228 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:15:04.989769 4367 tablet_service.cc:1467] Processing CreateTablet for tablet 485f583a08834f028aceec0eb882e29f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e66613dc8f6a472c8dbf38a1626afd0c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 06:15:04.989804 4507 tablet_service.cc:1467] Processing CreateTablet for tablet 485f583a08834f028aceec0eb882e29f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e66613dc8f6a472c8dbf38a1626afd0c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 06:15:04.989959 4367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 485f583a08834f028aceec0eb882e29f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:04.989971 4507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 485f583a08834f028aceec0eb882e29f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:04.990015 4647 tablet_service.cc:1467] Processing CreateTablet for tablet 485f583a08834f028aceec0eb882e29f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e66613dc8f6a472c8dbf38a1626afd0c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 06:15:04.990198 4647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 485f583a08834f028aceec0eb882e29f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:04.991329 4737 tablet_bootstrap.cc:492] T 485f583a08834f028aceec0eb882e29f P 9736545427474bbbabc55a4fcfb264de: Bootstrap starting.
I20260926 06:15:04.991581 4736 tablet_bootstrap.cc:492] T 485f583a08834f028aceec0eb882e29f P 20ab543b0b074455aae04921b83afa4e: Bootstrap starting.
I20260926 06:15:04.991616 4738 tablet_bootstrap.cc:492] T 485f583a08834f028aceec0eb882e29f P 19dab98ec7184b578b218bb35dcccc85: Bootstrap starting.
I20260926 06:15:04.991807 4737 tablet_bootstrap.cc:654] T 485f583a08834f028aceec0eb882e29f P 9736545427474bbbabc55a4fcfb264de: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:04.991942 4736 tablet_bootstrap.cc:654] T 485f583a08834f028aceec0eb882e29f P 20ab543b0b074455aae04921b83afa4e: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:04.992364 4738 tablet_bootstrap.cc:654] T 485f583a08834f028aceec0eb882e29f P 19dab98ec7184b578b218bb35dcccc85: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:04.992529 4737 tablet_bootstrap.cc:492] T 485f583a08834f028aceec0eb882e29f P 9736545427474bbbabc55a4fcfb264de: No bootstrap required, opened a new log
I20260926 06:15:04.992520 4736 tablet_bootstrap.cc:492] T 485f583a08834f028aceec0eb882e29f P 20ab543b0b074455aae04921b83afa4e: No bootstrap required, opened a new log
I20260926 06:15:04.992586 4736 ts_tablet_manager.cc:1397] T 485f583a08834f028aceec0eb882e29f P 20ab543b0b074455aae04921b83afa4e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:04.992595 4737 ts_tablet_manager.cc:1397] T 485f583a08834f028aceec0eb882e29f P 9736545427474bbbabc55a4fcfb264de: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:04.992766 4736 raft_consensus.cc:348] T 485f583a08834f028aceec0eb882e29f P 20ab543b0b074455aae04921b83afa4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } } peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } }
I20260926 06:15:04.992759 4737 raft_consensus.cc:348] T 485f583a08834f028aceec0eb882e29f P 9736545427474bbbabc55a4fcfb264de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } } peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } }
I20260926 06:15:04.992843 4737 raft_consensus.cc:374] T 485f583a08834f028aceec0eb882e29f P 9736545427474bbbabc55a4fcfb264de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:04.992843 4736 raft_consensus.cc:374] T 485f583a08834f028aceec0eb882e29f P 20ab543b0b074455aae04921b83afa4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:04.992866 4737 raft_consensus.cc:729] T 485f583a08834f028aceec0eb882e29f P 9736545427474bbbabc55a4fcfb264de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9736545427474bbbabc55a4fcfb264de, State: Initialized, Role: FOLLOWER
I20260926 06:15:04.992866 4736 raft_consensus.cc:729] T 485f583a08834f028aceec0eb882e29f P 20ab543b0b074455aae04921b83afa4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20ab543b0b074455aae04921b83afa4e, State: Initialized, Role: FOLLOWER
I20260926 06:15:04.992919 4737 consensus_queue.cc:260] T 485f583a08834f028aceec0eb882e29f P 9736545427474bbbabc55a4fcfb264de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } } peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } }
I20260926 06:15:04.992918 4736 consensus_queue.cc:260] T 485f583a08834f028aceec0eb882e29f P 20ab543b0b074455aae04921b83afa4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } } peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } }
I20260926 06:15:04.993013 4736 ts_tablet_manager.cc:1428] T 485f583a08834f028aceec0eb882e29f P 20ab543b0b074455aae04921b83afa4e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 06:15:04.993011 4737 ts_tablet_manager.cc:1428] T 485f583a08834f028aceec0eb882e29f P 9736545427474bbbabc55a4fcfb264de: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 06:15:04.993144 4744 tablet_replica.cc:439] T 485f583a08834f028aceec0eb882e29f P 9736545427474bbbabc55a4fcfb264de: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } } peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } } }
I20260926 06:15:04.993156 4743 tablet_replica.cc:439] T 485f583a08834f028aceec0eb882e29f P 20ab543b0b074455aae04921b83afa4e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } } peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } } }
I20260926 06:15:04.993221 4743 tablet_replica.cc:442] T 485f583a08834f028aceec0eb882e29f P 20ab543b0b074455aae04921b83afa4e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:15:04.993221 4744 tablet_replica.cc:442] T 485f583a08834f028aceec0eb882e29f P 9736545427474bbbabc55a4fcfb264de: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:15:04.993791 4738 tablet_bootstrap.cc:492] T 485f583a08834f028aceec0eb882e29f P 19dab98ec7184b578b218bb35dcccc85: No bootstrap required, opened a new log
I20260926 06:15:04.993853 4738 ts_tablet_manager.cc:1397] T 485f583a08834f028aceec0eb882e29f P 19dab98ec7184b578b218bb35dcccc85: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 06:15:04.994023 4738 raft_consensus.cc:348] T 485f583a08834f028aceec0eb882e29f P 19dab98ec7184b578b218bb35dcccc85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } } peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } }
I20260926 06:15:04.994112 4738 raft_consensus.cc:374] T 485f583a08834f028aceec0eb882e29f P 19dab98ec7184b578b218bb35dcccc85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:04.994150 4738 raft_consensus.cc:729] T 485f583a08834f028aceec0eb882e29f P 19dab98ec7184b578b218bb35dcccc85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19dab98ec7184b578b218bb35dcccc85, State: Initialized, Role: FOLLOWER
I20260926 06:15:04.994205 4738 consensus_queue.cc:260] T 485f583a08834f028aceec0eb882e29f P 19dab98ec7184b578b218bb35dcccc85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } } peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } }
I20260926 06:15:04.994304 4738 ts_tablet_manager.cc:1428] T 485f583a08834f028aceec0eb882e29f P 19dab98ec7184b578b218bb35dcccc85: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 06:15:04.994390 4742 tablet_replica.cc:439] T 485f583a08834f028aceec0eb882e29f P 19dab98ec7184b578b218bb35dcccc85: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } } peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } } }
I20260926 06:15:04.994467 4742 tablet_replica.cc:442] T 485f583a08834f028aceec0eb882e29f P 19dab98ec7184b578b218bb35dcccc85: This TxnStatusTablet replica's current role is: FOLLOWER
W20260926 06:15:04.998482 4714 tablet.cc:2369] T 165fd098bae848da9929ae1827f03962 P 9736545427474bbbabc55a4fcfb264de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:15:05.074249 4744 raft_consensus.cc:482] T 485f583a08834f028aceec0eb882e29f P 9736545427474bbbabc55a4fcfb264de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:15:05.074352 4744 raft_consensus.cc:504] T 485f583a08834f028aceec0eb882e29f P 9736545427474bbbabc55a4fcfb264de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } } peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } }
I20260926 06:15:05.074522 4744 leader_election.cc:288] T 485f583a08834f028aceec0eb882e29f P 9736545427474bbbabc55a4fcfb264de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20ab543b0b074455aae04921b83afa4e (127.2.110.193:46297), 19dab98ec7184b578b218bb35dcccc85 (127.2.110.194:45967)
I20260926 06:15:05.074973 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "485f583a08834f028aceec0eb882e29f" candidate_uuid: "9736545427474bbbabc55a4fcfb264de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19dab98ec7184b578b218bb35dcccc85" is_pre_election: true
I20260926 06:15:05.075017 4387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "485f583a08834f028aceec0eb882e29f" candidate_uuid: "9736545427474bbbabc55a4fcfb264de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20ab543b0b074455aae04921b83afa4e" is_pre_election: true
I20260926 06:15:05.075158 4387 raft_consensus.cc:2454] T 485f583a08834f028aceec0eb882e29f P 20ab543b0b074455aae04921b83afa4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9736545427474bbbabc55a4fcfb264de in term 0.
I20260926 06:15:05.075158 4527 raft_consensus.cc:2454] T 485f583a08834f028aceec0eb882e29f P 19dab98ec7184b578b218bb35dcccc85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9736545427474bbbabc55a4fcfb264de in term 0.
I20260926 06:15:05.075424 4600 leader_election.cc:302] T 485f583a08834f028aceec0eb882e29f P 9736545427474bbbabc55a4fcfb264de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20ab543b0b074455aae04921b83afa4e, 9736545427474bbbabc55a4fcfb264de; no voters:
I20260926 06:15:05.075546 4744 raft_consensus.cc:2781] T 485f583a08834f028aceec0eb882e29f P 9736545427474bbbabc55a4fcfb264de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:15:05.075589 4744 raft_consensus.cc:482] T 485f583a08834f028aceec0eb882e29f P 9736545427474bbbabc55a4fcfb264de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:15:05.075611 4744 raft_consensus.cc:3037] T 485f583a08834f028aceec0eb882e29f P 9736545427474bbbabc55a4fcfb264de [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:05.076275 4744 raft_consensus.cc:504] T 485f583a08834f028aceec0eb882e29f P 9736545427474bbbabc55a4fcfb264de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } } peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } }
I20260926 06:15:05.076424 4744 leader_election.cc:288] T 485f583a08834f028aceec0eb882e29f P 9736545427474bbbabc55a4fcfb264de [CANDIDATE]: Term 1 election: Requested vote from peers 20ab543b0b074455aae04921b83afa4e (127.2.110.193:46297), 19dab98ec7184b578b218bb35dcccc85 (127.2.110.194:45967)
I20260926 06:15:05.076607 4387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "485f583a08834f028aceec0eb882e29f" candidate_uuid: "9736545427474bbbabc55a4fcfb264de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20ab543b0b074455aae04921b83afa4e"
I20260926 06:15:05.076646 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "485f583a08834f028aceec0eb882e29f" candidate_uuid: "9736545427474bbbabc55a4fcfb264de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19dab98ec7184b578b218bb35dcccc85"
I20260926 06:15:05.076735 4387 raft_consensus.cc:3037] T 485f583a08834f028aceec0eb882e29f P 20ab543b0b074455aae04921b83afa4e [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:05.076733 4527 raft_consensus.cc:3037] T 485f583a08834f028aceec0eb882e29f P 19dab98ec7184b578b218bb35dcccc85 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:05.077421 4527 raft_consensus.cc:2454] T 485f583a08834f028aceec0eb882e29f P 19dab98ec7184b578b218bb35dcccc85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9736545427474bbbabc55a4fcfb264de in term 1.
I20260926 06:15:05.077441 4387 raft_consensus.cc:2454] T 485f583a08834f028aceec0eb882e29f P 20ab543b0b074455aae04921b83afa4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9736545427474bbbabc55a4fcfb264de in term 1.
I20260926 06:15:05.077603 4599 leader_election.cc:302] T 485f583a08834f028aceec0eb882e29f P 9736545427474bbbabc55a4fcfb264de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19dab98ec7184b578b218bb35dcccc85, 9736545427474bbbabc55a4fcfb264de; no voters:
I20260926 06:15:05.077734 4744 raft_consensus.cc:2781] T 485f583a08834f028aceec0eb882e29f P 9736545427474bbbabc55a4fcfb264de [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:15:05.077797 4744 raft_consensus.cc:686] T 485f583a08834f028aceec0eb882e29f P 9736545427474bbbabc55a4fcfb264de [term 1 LEADER]: Becoming Leader. State: Replica: 9736545427474bbbabc55a4fcfb264de, State: Running, Role: LEADER
I20260926 06:15:05.077849 4744 consensus_queue.cc:237] T 485f583a08834f028aceec0eb882e29f P 9736545427474bbbabc55a4fcfb264de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } } peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } }
I20260926 06:15:05.078171 4746 tablet_replica.cc:439] T 485f583a08834f028aceec0eb882e29f P 9736545427474bbbabc55a4fcfb264de: TxnStatusTablet state changed. Reason: New leader 9736545427474bbbabc55a4fcfb264de. Latest consensus state: current_term: 1 leader_uuid: "9736545427474bbbabc55a4fcfb264de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } } peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } } }
I20260926 06:15:05.078261 4746 tablet_replica.cc:442] T 485f583a08834f028aceec0eb882e29f P 9736545427474bbbabc55a4fcfb264de: This TxnStatusTablet replica's current role is: LEADER
I20260926 06:15:05.078459 4758 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 06:15:05.078606 4758 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 06:15:05.078536 4230 catalog_manager.cc:5495] T 485f583a08834f028aceec0eb882e29f P 9736545427474bbbabc55a4fcfb264de reported cstate change: term changed from 0 to 1, leader changed from <none> to 9736545427474bbbabc55a4fcfb264de (127.2.110.195). New cstate: current_term: 1 leader_uuid: "9736545427474bbbabc55a4fcfb264de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } health_report { overall_health: HEALTHY } } }
I20260926 06:15:05.091715 4734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:05.088703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59592 (local address 127.2.110.195:36447)
0926 06:15:05.088764 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:05.088765 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:05.088888 (+ 123us) server_negotiation.cc:407] Connection header received
0926 06:15:05.088977 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:05.088978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:05.088993 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:05.089049 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:15:05.089168 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:05.089728 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:05.090876 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:05.091073 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:05.091140 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:05.091327 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:15:05.091452 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:15:05.091529 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:05.091586 (+ 57us) server_negotiation.cc:299] Negotiation successful
0926 06:15:05.091611 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 06:15:05.091743 4753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:05.088604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.195:36447 (local address 127.0.0.1:59592)
0926 06:15:05.088793 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:05.088813 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:05.088897 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:05.089053 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:05.089053 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:05.089055 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 06:15:05.089111 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:05.089114 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:05.089817 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:05.089818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:05.090777 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:05.090780 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:05.091149 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:05.091151 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:05.091224 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:05.091243 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 06:15:05.091520 (+ 277us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 06:15:05.091528 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 06:15:05.091583 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 06:15:05.091656 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143}
I20260926 06:15:05.093659 4527 raft_consensus.cc:1261] T 485f583a08834f028aceec0eb882e29f P 19dab98ec7184b578b218bb35dcccc85 [term 1 FOLLOWER]: Refusing update from remote peer 9736545427474bbbabc55a4fcfb264de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:15:05.093690 4387 raft_consensus.cc:1261] T 485f583a08834f028aceec0eb882e29f P 20ab543b0b074455aae04921b83afa4e [term 1 FOLLOWER]: Refusing update from remote peer 9736545427474bbbabc55a4fcfb264de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:15:05.094012 4746 consensus_queue.cc:1035] T 485f583a08834f028aceec0eb882e29f P 9736545427474bbbabc55a4fcfb264de [LEADER]: Connected to new peer: Peer: permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:15:05.094144 4744 consensus_queue.cc:1035] T 485f583a08834f028aceec0eb882e29f P 9736545427474bbbabc55a4fcfb264de [LEADER]: Connected to new peer: Peer: permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:15:05.095329 4743 tablet_replica.cc:439] T 485f583a08834f028aceec0eb882e29f P 20ab543b0b074455aae04921b83afa4e: TxnStatusTablet state changed. Reason: New leader 9736545427474bbbabc55a4fcfb264de. Latest consensus state: current_term: 1 leader_uuid: "9736545427474bbbabc55a4fcfb264de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } } peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } } }
I20260926 06:15:05.095430 4743 tablet_replica.cc:442] T 485f583a08834f028aceec0eb882e29f P 20ab543b0b074455aae04921b83afa4e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:15:05.095733 4742 tablet_replica.cc:439] T 485f583a08834f028aceec0eb882e29f P 19dab98ec7184b578b218bb35dcccc85: TxnStatusTablet state changed. Reason: New leader 9736545427474bbbabc55a4fcfb264de. Latest consensus state: current_term: 1 leader_uuid: "9736545427474bbbabc55a4fcfb264de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } } peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } } }
I20260926 06:15:05.095819 4742 tablet_replica.cc:442] T 485f583a08834f028aceec0eb882e29f P 19dab98ec7184b578b218bb35dcccc85: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:15:05.096479 4744 tablet_replica.cc:439] T 485f583a08834f028aceec0eb882e29f P 9736545427474bbbabc55a4fcfb264de: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9736545427474bbbabc55a4fcfb264de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } } peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } } }
I20260926 06:15:05.096580 4744 tablet_replica.cc:442] T 485f583a08834f028aceec0eb882e29f P 9736545427474bbbabc55a4fcfb264de: This TxnStatusTablet replica's current role is: LEADER
I20260926 06:15:05.096644 4761 mvcc.cc:205] Tried to move back new op lower bound from 7333491937661104128 to 7333491937599201280. Current Snapshot: MvccSnapshot[applied={T|T < 7333491937661104128}]
I20260926 06:15:05.096699 4742 tablet_replica.cc:439] T 485f583a08834f028aceec0eb882e29f P 19dab98ec7184b578b218bb35dcccc85: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9736545427474bbbabc55a4fcfb264de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } } peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } } }
I20260926 06:15:05.096771 4742 tablet_replica.cc:442] T 485f583a08834f028aceec0eb882e29f P 19dab98ec7184b578b218bb35dcccc85: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:15:05.097590 4744 tablet_replica.cc:439] T 485f583a08834f028aceec0eb882e29f P 9736545427474bbbabc55a4fcfb264de: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9736545427474bbbabc55a4fcfb264de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } } peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } } }
I20260926 06:15:05.097684 4744 tablet_replica.cc:442] T 485f583a08834f028aceec0eb882e29f P 9736545427474bbbabc55a4fcfb264de: This TxnStatusTablet replica's current role is: LEADER
I20260926 06:15:05.098156 4743 tablet_replica.cc:439] T 485f583a08834f028aceec0eb882e29f P 20ab543b0b074455aae04921b83afa4e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9736545427474bbbabc55a4fcfb264de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } } peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } } }
I20260926 06:15:05.098251 4743 tablet_replica.cc:442] T 485f583a08834f028aceec0eb882e29f P 20ab543b0b074455aae04921b83afa4e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:15:05.098526 4763 mvcc.cc:205] Tried to move back new op lower bound from 7333491937661104128 to 7333491937599201280. Current Snapshot: MvccSnapshot[applied={T|T < 7333491937661104128 or (T in {7333491937661104128})}]
I20260926 06:15:05.099046 4762 mvcc.cc:205] Tried to move back new op lower bound from 7333491937661104128 to 7333491937599201280. Current Snapshot: MvccSnapshot[applied={T|T < 7333491937661104128 or (T in {7333491937661104128})}]
I20260926 06:15:05.102316 4734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:05.099630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59594 (local address 127.2.110.195:36447)
0926 06:15:05.099668 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:05.099670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:05.099679 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:15:05.099731 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:05.099731 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:05.099740 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:05.099787 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:15:05.099911 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:05.100361 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:05.101516 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:05.101821 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:05.101889 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:05.102068 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:15:05.102107 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:15:05.102159 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:05.102214 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 06:15:05.102236 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 06:15:05.107548 4734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:05.105005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:35299 (local address 127.2.110.195:36447)
0926 06:15:05.105143 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:05.105144 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:05.105153 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:15:05.105172 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:05.105173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:05.105182 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:05.105252 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:15:05.105413 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:05.105920 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:05.106766 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:05.106976 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:05.107065 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:05.107265 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:15:05.107302 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:15:05.107366 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:05.107431 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 06:15:05.107450 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":110}
I20260926 06:15:05.107559 4719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:05.104952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.195:36447 (local address 127.2.110.195:35299)
0926 06:15:05.105014 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:05.105025 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:05.105069 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:05.105254 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:05.105255 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:05.105256 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 06:15:05.105332 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:05.105358 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:05.106004 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:05.106004 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:05.106658 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:05.106660 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:05.107065 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:05.107066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:05.107131 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:05.107153 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 06:15:05.107364 (+ 211us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 06:15:05.107366 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 06:15:05.107427 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 06:15:05.107469 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":2}
I20260926 06:15:05.110560 4387 raft_consensus.cc:1261] T 165fd098bae848da9929ae1827f03962 P 20ab543b0b074455aae04921b83afa4e [term 1 FOLLOWER]: Refusing update from remote peer 9736545427474bbbabc55a4fcfb264de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:15:05.110805 4527 raft_consensus.cc:1261] T 165fd098bae848da9929ae1827f03962 P 19dab98ec7184b578b218bb35dcccc85 [term 1 FOLLOWER]: Refusing update from remote peer 9736545427474bbbabc55a4fcfb264de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:15:05.110951 4768 consensus_queue.cc:1035] T 165fd098bae848da9929ae1827f03962 P 9736545427474bbbabc55a4fcfb264de [LEADER]: Connected to new peer: Peer: permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:15:05.111265 4746 consensus_queue.cc:1035] T 165fd098bae848da9929ae1827f03962 P 9736545427474bbbabc55a4fcfb264de [LEADER]: Connected to new peer: Peer: permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:15:05.130627 4230 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:15:05.130764 4230 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:15:05.131829 4230 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9fdc65b166ea49feab076956712a2657: Sending DeleteTablet for 3 replicas of tablet 165fd098bae848da9929ae1827f03962
I20260926 06:15:05.132172 4367 tablet_service.cc:1514] Processing DeleteTablet for tablet 165fd098bae848da9929ae1827f03962 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:15:05 UTC) from {username='kudu', principal='kudu/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:35210
I20260926 06:15:05.132210 4646 tablet_service.cc:1514] Processing DeleteTablet for tablet 165fd098bae848da9929ae1827f03962 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:15:05 UTC) from {username='kudu', principal='kudu/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:59584
I20260926 06:15:05.132493 4230 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47178:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:15:05.132647 4230 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:15:05.132681 4776 tablet_replica.cc:330] T 165fd098bae848da9929ae1827f03962 P 20ab543b0b074455aae04921b83afa4e: stopping tablet replica
I20260926 06:15:05.132647 4507 tablet_service.cc:1514] Processing DeleteTablet for tablet 165fd098bae848da9929ae1827f03962 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:15:05 UTC) from {username='kudu', principal='kudu/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:32984
I20260926 06:15:05.132802 4776 raft_consensus.cc:2229] T 165fd098bae848da9929ae1827f03962 P 20ab543b0b074455aae04921b83afa4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:15:05.132880 4776 raft_consensus.cc:2258] T 165fd098bae848da9929ae1827f03962 P 20ab543b0b074455aae04921b83afa4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:15:05.133412 4776 ts_tablet_manager.cc:1905] T 165fd098bae848da9929ae1827f03962 P 20ab543b0b074455aae04921b83afa4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:15:05.133868 4367 tablet_service.cc:1467] Processing CreateTablet for tablet 85acd211fa864ca6a16fbea04f2fb20c (DEFAULT_TABLE table=test-table [id=2bea85a2f591465c8bab8a714a71e795]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:15:05.133977 4367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85acd211fa864ca6a16fbea04f2fb20c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:05.133958 4507 tablet_service.cc:1467] Processing CreateTablet for tablet 85acd211fa864ca6a16fbea04f2fb20c (DEFAULT_TABLE table=test-table [id=2bea85a2f591465c8bab8a714a71e795]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:15:05.134061 4507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85acd211fa864ca6a16fbea04f2fb20c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:05.134294 4646 tablet_service.cc:1467] Processing CreateTablet for tablet 85acd211fa864ca6a16fbea04f2fb20c (DEFAULT_TABLE table=test-table [id=2bea85a2f591465c8bab8a714a71e795]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:15:05.134407 4646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85acd211fa864ca6a16fbea04f2fb20c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:05.135389 4736 tablet_bootstrap.cc:492] T 85acd211fa864ca6a16fbea04f2fb20c P 20ab543b0b074455aae04921b83afa4e: Bootstrap starting.
I20260926 06:15:05.135775 4736 tablet_bootstrap.cc:654] T 85acd211fa864ca6a16fbea04f2fb20c P 20ab543b0b074455aae04921b83afa4e: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:05.136152 4738 tablet_bootstrap.cc:492] T 85acd211fa864ca6a16fbea04f2fb20c P 19dab98ec7184b578b218bb35dcccc85: Bootstrap starting.
I20260926 06:15:05.136459 4736 tablet_bootstrap.cc:492] T 85acd211fa864ca6a16fbea04f2fb20c P 20ab543b0b074455aae04921b83afa4e: No bootstrap required, opened a new log
I20260926 06:15:05.136521 4736 ts_tablet_manager.cc:1397] T 85acd211fa864ca6a16fbea04f2fb20c P 20ab543b0b074455aae04921b83afa4e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:05.136590 4738 tablet_bootstrap.cc:654] T 85acd211fa864ca6a16fbea04f2fb20c P 19dab98ec7184b578b218bb35dcccc85: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:05.136667 4736 raft_consensus.cc:348] T 85acd211fa864ca6a16fbea04f2fb20c P 20ab543b0b074455aae04921b83afa4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } } peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } }
I20260926 06:15:05.136755 4736 raft_consensus.cc:374] T 85acd211fa864ca6a16fbea04f2fb20c P 20ab543b0b074455aae04921b83afa4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:05.136782 4736 raft_consensus.cc:729] T 85acd211fa864ca6a16fbea04f2fb20c P 20ab543b0b074455aae04921b83afa4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20ab543b0b074455aae04921b83afa4e, State: Initialized, Role: FOLLOWER
I20260926 06:15:05.136824 4736 consensus_queue.cc:260] T 85acd211fa864ca6a16fbea04f2fb20c P 20ab543b0b074455aae04921b83afa4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } } peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } }
I20260926 06:15:05.136917 4736 ts_tablet_manager.cc:1428] T 85acd211fa864ca6a16fbea04f2fb20c P 20ab543b0b074455aae04921b83afa4e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 06:15:05.136950 4737 tablet_bootstrap.cc:492] T 85acd211fa864ca6a16fbea04f2fb20c P 9736545427474bbbabc55a4fcfb264de: Bootstrap starting.
I20260926 06:15:05.137362 4737 tablet_bootstrap.cc:654] T 85acd211fa864ca6a16fbea04f2fb20c P 9736545427474bbbabc55a4fcfb264de: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:05.138077 4737 tablet_bootstrap.cc:492] T 85acd211fa864ca6a16fbea04f2fb20c P 9736545427474bbbabc55a4fcfb264de: No bootstrap required, opened a new log
I20260926 06:15:05.138126 4737 ts_tablet_manager.cc:1397] T 85acd211fa864ca6a16fbea04f2fb20c P 9736545427474bbbabc55a4fcfb264de: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 06:15:05.138288 4737 raft_consensus.cc:348] T 85acd211fa864ca6a16fbea04f2fb20c P 9736545427474bbbabc55a4fcfb264de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } } peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } }
I20260926 06:15:05.138362 4737 raft_consensus.cc:374] T 85acd211fa864ca6a16fbea04f2fb20c P 9736545427474bbbabc55a4fcfb264de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:05.138383 4737 raft_consensus.cc:729] T 85acd211fa864ca6a16fbea04f2fb20c P 9736545427474bbbabc55a4fcfb264de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9736545427474bbbabc55a4fcfb264de, State: Initialized, Role: FOLLOWER
I20260926 06:15:05.138425 4737 consensus_queue.cc:260] T 85acd211fa864ca6a16fbea04f2fb20c P 9736545427474bbbabc55a4fcfb264de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } } peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } }
I20260926 06:15:05.138514 4737 ts_tablet_manager.cc:1428] T 85acd211fa864ca6a16fbea04f2fb20c P 9736545427474bbbabc55a4fcfb264de: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:05.138873 4777 tablet_replica.cc:330] T 165fd098bae848da9929ae1827f03962 P 9736545427474bbbabc55a4fcfb264de: stopping tablet replica
I20260926 06:15:05.138998 4777 raft_consensus.cc:2229] T 165fd098bae848da9929ae1827f03962 P 9736545427474bbbabc55a4fcfb264de [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:15:05.139035 4776 ts_tablet_manager.cc:1918] T 165fd098bae848da9929ae1827f03962 P 20ab543b0b074455aae04921b83afa4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 06:15:05.139106 4776 log.cc:1196] T 165fd098bae848da9929ae1827f03962 P 20ab543b0b074455aae04921b83afa4e: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-0/wal/wals/165fd098bae848da9929ae1827f03962
I20260926 06:15:05.139159 4777 raft_consensus.cc:2258] T 165fd098bae848da9929ae1827f03962 P 9736545427474bbbabc55a4fcfb264de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:15:05.139389 4776 ts_tablet_manager.cc:1939] T 165fd098bae848da9929ae1827f03962 P 20ab543b0b074455aae04921b83afa4e: Deleting consensus metadata
I20260926 06:15:05.139755 4778 tablet_replica.cc:330] T 165fd098bae848da9929ae1827f03962 P 19dab98ec7184b578b218bb35dcccc85: stopping tablet replica
I20260926 06:15:05.139779 4777 ts_tablet_manager.cc:1905] T 165fd098bae848da9929ae1827f03962 P 9736545427474bbbabc55a4fcfb264de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:15:05.139834 4216 catalog_manager.cc:4841] TS 20ab543b0b074455aae04921b83afa4e (127.2.110.193:46297): tablet 165fd098bae848da9929ae1827f03962 (table test-table [id=7f74433c00134b2286e5edb2f3f003a5]) successfully deleted
I20260926 06:15:05.139878 4778 raft_consensus.cc:2229] T 165fd098bae848da9929ae1827f03962 P 19dab98ec7184b578b218bb35dcccc85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:15:05.139937 4778 raft_consensus.cc:2258] T 165fd098bae848da9929ae1827f03962 P 19dab98ec7184b578b218bb35dcccc85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:15:05.140309 4778 ts_tablet_manager.cc:1905] T 165fd098bae848da9929ae1827f03962 P 19dab98ec7184b578b218bb35dcccc85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:15:05.141561 4738 tablet_bootstrap.cc:492] T 85acd211fa864ca6a16fbea04f2fb20c P 19dab98ec7184b578b218bb35dcccc85: No bootstrap required, opened a new log
I20260926 06:15:05.141633 4738 ts_tablet_manager.cc:1397] T 85acd211fa864ca6a16fbea04f2fb20c P 19dab98ec7184b578b218bb35dcccc85: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260926 06:15:05.141744 4778 ts_tablet_manager.cc:1918] T 165fd098bae848da9929ae1827f03962 P 19dab98ec7184b578b218bb35dcccc85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 06:15:05.141815 4778 log.cc:1196] T 165fd098bae848da9929ae1827f03962 P 19dab98ec7184b578b218bb35dcccc85: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-1/wal/wals/165fd098bae848da9929ae1827f03962
I20260926 06:15:05.141776 4738 raft_consensus.cc:348] T 85acd211fa864ca6a16fbea04f2fb20c P 19dab98ec7184b578b218bb35dcccc85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } } peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } }
I20260926 06:15:05.141875 4738 raft_consensus.cc:374] T 85acd211fa864ca6a16fbea04f2fb20c P 19dab98ec7184b578b218bb35dcccc85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:05.141901 4738 raft_consensus.cc:729] T 85acd211fa864ca6a16fbea04f2fb20c P 19dab98ec7184b578b218bb35dcccc85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19dab98ec7184b578b218bb35dcccc85, State: Initialized, Role: FOLLOWER
I20260926 06:15:05.141943 4738 consensus_queue.cc:260] T 85acd211fa864ca6a16fbea04f2fb20c P 19dab98ec7184b578b218bb35dcccc85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } } peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } }
I20260926 06:15:05.142031 4738 ts_tablet_manager.cc:1428] T 85acd211fa864ca6a16fbea04f2fb20c P 19dab98ec7184b578b218bb35dcccc85: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 06:15:05.142091 4778 ts_tablet_manager.cc:1939] T 165fd098bae848da9929ae1827f03962 P 19dab98ec7184b578b218bb35dcccc85: Deleting consensus metadata
I20260926 06:15:05.141400 4777 ts_tablet_manager.cc:1918] T 165fd098bae848da9929ae1827f03962 P 9736545427474bbbabc55a4fcfb264de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 06:15:05.142283 4777 log.cc:1196] T 165fd098bae848da9929ae1827f03962 P 9736545427474bbbabc55a4fcfb264de: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-2/wal/wals/165fd098bae848da9929ae1827f03962
I20260926 06:15:05.142527 4215 catalog_manager.cc:4841] TS 19dab98ec7184b578b218bb35dcccc85 (127.2.110.194:45967): tablet 165fd098bae848da9929ae1827f03962 (table test-table [id=7f74433c00134b2286e5edb2f3f003a5]) successfully deleted
I20260926 06:15:05.142755 4777 ts_tablet_manager.cc:1939] T 165fd098bae848da9929ae1827f03962 P 9736545427474bbbabc55a4fcfb264de: Deleting consensus metadata
I20260926 06:15:05.143117 4217 catalog_manager.cc:4841] TS 9736545427474bbbabc55a4fcfb264de (127.2.110.195:36447): tablet 165fd098bae848da9929ae1827f03962 (table test-table [id=7f74433c00134b2286e5edb2f3f003a5]) successfully deleted
I20260926 06:15:05.224921 4744 raft_consensus.cc:482] T 85acd211fa864ca6a16fbea04f2fb20c P 9736545427474bbbabc55a4fcfb264de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:15:05.225064 4744 raft_consensus.cc:504] T 85acd211fa864ca6a16fbea04f2fb20c P 9736545427474bbbabc55a4fcfb264de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } } peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } }
I20260926 06:15:05.225230 4744 leader_election.cc:288] T 85acd211fa864ca6a16fbea04f2fb20c P 9736545427474bbbabc55a4fcfb264de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20ab543b0b074455aae04921b83afa4e (127.2.110.193:46297), 19dab98ec7184b578b218bb35dcccc85 (127.2.110.194:45967)
I20260926 06:15:05.225497 4387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85acd211fa864ca6a16fbea04f2fb20c" candidate_uuid: "9736545427474bbbabc55a4fcfb264de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20ab543b0b074455aae04921b83afa4e" is_pre_election: true
I20260926 06:15:05.225500 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85acd211fa864ca6a16fbea04f2fb20c" candidate_uuid: "9736545427474bbbabc55a4fcfb264de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19dab98ec7184b578b218bb35dcccc85" is_pre_election: true
I20260926 06:15:05.225593 4527 raft_consensus.cc:2454] T 85acd211fa864ca6a16fbea04f2fb20c P 19dab98ec7184b578b218bb35dcccc85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9736545427474bbbabc55a4fcfb264de in term 0.
I20260926 06:15:05.225593 4387 raft_consensus.cc:2454] T 85acd211fa864ca6a16fbea04f2fb20c P 20ab543b0b074455aae04921b83afa4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9736545427474bbbabc55a4fcfb264de in term 0.
I20260926 06:15:05.225752 4600 leader_election.cc:302] T 85acd211fa864ca6a16fbea04f2fb20c P 9736545427474bbbabc55a4fcfb264de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20ab543b0b074455aae04921b83afa4e, 9736545427474bbbabc55a4fcfb264de; no voters:
I20260926 06:15:05.225843 4744 raft_consensus.cc:2781] T 85acd211fa864ca6a16fbea04f2fb20c P 9736545427474bbbabc55a4fcfb264de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:15:05.225880 4744 raft_consensus.cc:482] T 85acd211fa864ca6a16fbea04f2fb20c P 9736545427474bbbabc55a4fcfb264de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:15:05.225907 4744 raft_consensus.cc:3037] T 85acd211fa864ca6a16fbea04f2fb20c P 9736545427474bbbabc55a4fcfb264de [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:05.226531 4744 raft_consensus.cc:504] T 85acd211fa864ca6a16fbea04f2fb20c P 9736545427474bbbabc55a4fcfb264de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } } peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } }
I20260926 06:15:05.226677 4744 leader_election.cc:288] T 85acd211fa864ca6a16fbea04f2fb20c P 9736545427474bbbabc55a4fcfb264de [CANDIDATE]: Term 1 election: Requested vote from peers 20ab543b0b074455aae04921b83afa4e (127.2.110.193:46297), 19dab98ec7184b578b218bb35dcccc85 (127.2.110.194:45967)
I20260926 06:15:05.226811 4387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85acd211fa864ca6a16fbea04f2fb20c" candidate_uuid: "9736545427474bbbabc55a4fcfb264de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20ab543b0b074455aae04921b83afa4e"
I20260926 06:15:05.226892 4387 raft_consensus.cc:3037] T 85acd211fa864ca6a16fbea04f2fb20c P 20ab543b0b074455aae04921b83afa4e [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:05.226878 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85acd211fa864ca6a16fbea04f2fb20c" candidate_uuid: "9736545427474bbbabc55a4fcfb264de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19dab98ec7184b578b218bb35dcccc85"
I20260926 06:15:05.226984 4527 raft_consensus.cc:3037] T 85acd211fa864ca6a16fbea04f2fb20c P 19dab98ec7184b578b218bb35dcccc85 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:05.227500 4387 raft_consensus.cc:2454] T 85acd211fa864ca6a16fbea04f2fb20c P 20ab543b0b074455aae04921b83afa4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9736545427474bbbabc55a4fcfb264de in term 1.
I20260926 06:15:05.227677 4600 leader_election.cc:302] T 85acd211fa864ca6a16fbea04f2fb20c P 9736545427474bbbabc55a4fcfb264de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20ab543b0b074455aae04921b83afa4e, 9736545427474bbbabc55a4fcfb264de; no voters:
I20260926 06:15:05.227774 4744 raft_consensus.cc:2781] T 85acd211fa864ca6a16fbea04f2fb20c P 9736545427474bbbabc55a4fcfb264de [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:15:05.227818 4744 raft_consensus.cc:686] T 85acd211fa864ca6a16fbea04f2fb20c P 9736545427474bbbabc55a4fcfb264de [term 1 LEADER]: Becoming Leader. State: Replica: 9736545427474bbbabc55a4fcfb264de, State: Running, Role: LEADER
I20260926 06:15:05.227875 4744 consensus_queue.cc:237] T 85acd211fa864ca6a16fbea04f2fb20c P 9736545427474bbbabc55a4fcfb264de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } } peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } }
I20260926 06:15:05.227886 4527 raft_consensus.cc:2454] T 85acd211fa864ca6a16fbea04f2fb20c P 19dab98ec7184b578b218bb35dcccc85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9736545427474bbbabc55a4fcfb264de in term 1.
I20260926 06:15:05.228432 4230 catalog_manager.cc:5495] T 85acd211fa864ca6a16fbea04f2fb20c P 9736545427474bbbabc55a4fcfb264de reported cstate change: term changed from 0 to 1, leader changed from <none> to 9736545427474bbbabc55a4fcfb264de (127.2.110.195). New cstate: current_term: 1 leader_uuid: "9736545427474bbbabc55a4fcfb264de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } health_report { overall_health: HEALTHY } } }
I20260926 06:15:05.235731 4527 raft_consensus.cc:1261] T 85acd211fa864ca6a16fbea04f2fb20c P 19dab98ec7184b578b218bb35dcccc85 [term 1 FOLLOWER]: Refusing update from remote peer 9736545427474bbbabc55a4fcfb264de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:15:05.235797 4387 raft_consensus.cc:1261] T 85acd211fa864ca6a16fbea04f2fb20c P 20ab543b0b074455aae04921b83afa4e [term 1 FOLLOWER]: Refusing update from remote peer 9736545427474bbbabc55a4fcfb264de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:15:05.235954 4746 consensus_queue.cc:1035] T 85acd211fa864ca6a16fbea04f2fb20c P 9736545427474bbbabc55a4fcfb264de [LEADER]: Connected to new peer: Peer: permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:15:05.236054 4768 consensus_queue.cc:1035] T 85acd211fa864ca6a16fbea04f2fb20c P 9736545427474bbbabc55a4fcfb264de [LEADER]: Connected to new peer: Peer: permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 06:15:05.240777 4761 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 06:15:06.762985 4228 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:15:06.763132 4228 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:15:06.764027 4228 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9fdc65b166ea49feab076956712a2657: Sending DeleteTablet for 3 replicas of tablet 85acd211fa864ca6a16fbea04f2fb20c
I20260926 06:15:06.764287 4367 tablet_service.cc:1514] Processing DeleteTablet for tablet 85acd211fa864ca6a16fbea04f2fb20c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:15:06 UTC) from {username='kudu', principal='kudu/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:35210
I20260926 06:15:06.764282 4646 tablet_service.cc:1514] Processing DeleteTablet for tablet 85acd211fa864ca6a16fbea04f2fb20c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:15:06 UTC) from {username='kudu', principal='kudu/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:59584
I20260926 06:15:06.764346 4507 tablet_service.cc:1514] Processing DeleteTablet for tablet 85acd211fa864ca6a16fbea04f2fb20c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:15:06 UTC) from {username='kudu', principal='kudu/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:32984
I20260926 06:15:06.764518 4805 tablet_replica.cc:330] T 85acd211fa864ca6a16fbea04f2fb20c P 20ab543b0b074455aae04921b83afa4e: stopping tablet replica
I20260926 06:15:06.764585 4805 raft_consensus.cc:2229] T 85acd211fa864ca6a16fbea04f2fb20c P 20ab543b0b074455aae04921b83afa4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:15:06.764631 4805 raft_consensus.cc:2258] T 85acd211fa864ca6a16fbea04f2fb20c P 20ab543b0b074455aae04921b83afa4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:15:06.764616 4807 tablet_replica.cc:330] T 85acd211fa864ca6a16fbea04f2fb20c P 19dab98ec7184b578b218bb35dcccc85: stopping tablet replica
I20260926 06:15:06.764734 4806 tablet_replica.cc:330] T 85acd211fa864ca6a16fbea04f2fb20c P 9736545427474bbbabc55a4fcfb264de: stopping tablet replica
I20260926 06:15:06.764793 4806 raft_consensus.cc:2229] T 85acd211fa864ca6a16fbea04f2fb20c P 9736545427474bbbabc55a4fcfb264de [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:15:06.764794 4807 raft_consensus.cc:2229] T 85acd211fa864ca6a16fbea04f2fb20c P 19dab98ec7184b578b218bb35dcccc85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:15:06.764842 4807 raft_consensus.cc:2258] T 85acd211fa864ca6a16fbea04f2fb20c P 19dab98ec7184b578b218bb35dcccc85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:15:06.764865 4806 raft_consensus.cc:2258] T 85acd211fa864ca6a16fbea04f2fb20c P 9736545427474bbbabc55a4fcfb264de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:15:06.765021 4805 ts_tablet_manager.cc:1905] T 85acd211fa864ca6a16fbea04f2fb20c P 20ab543b0b074455aae04921b83afa4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:15:06.765061 4806 ts_tablet_manager.cc:1905] T 85acd211fa864ca6a16fbea04f2fb20c P 9736545427474bbbabc55a4fcfb264de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:15:06.766841 4806 ts_tablet_manager.cc:1918] T 85acd211fa864ca6a16fbea04f2fb20c P 9736545427474bbbabc55a4fcfb264de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 06:15:06.766841 4805 ts_tablet_manager.cc:1918] T 85acd211fa864ca6a16fbea04f2fb20c P 20ab543b0b074455aae04921b83afa4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 06:15:06.766906 4806 log.cc:1196] T 85acd211fa864ca6a16fbea04f2fb20c P 9736545427474bbbabc55a4fcfb264de: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-2/wal/wals/85acd211fa864ca6a16fbea04f2fb20c
I20260926 06:15:06.766956 4805 log.cc:1196] T 85acd211fa864ca6a16fbea04f2fb20c P 20ab543b0b074455aae04921b83afa4e: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-0/wal/wals/85acd211fa864ca6a16fbea04f2fb20c
I20260926 06:15:06.767176 4806 ts_tablet_manager.cc:1939] T 85acd211fa864ca6a16fbea04f2fb20c P 9736545427474bbbabc55a4fcfb264de: Deleting consensus metadata
I20260926 06:15:06.767200 4805 ts_tablet_manager.cc:1939] T 85acd211fa864ca6a16fbea04f2fb20c P 20ab543b0b074455aae04921b83afa4e: Deleting consensus metadata
I20260926 06:15:06.767666 4217 catalog_manager.cc:4841] TS 9736545427474bbbabc55a4fcfb264de (127.2.110.195:36447): tablet 85acd211fa864ca6a16fbea04f2fb20c (table test-table [id=2bea85a2f591465c8bab8a714a71e795]) successfully deleted
I20260926 06:15:06.767675 4216 catalog_manager.cc:4841] TS 20ab543b0b074455aae04921b83afa4e (127.2.110.193:46297): tablet 85acd211fa864ca6a16fbea04f2fb20c (table test-table [id=2bea85a2f591465c8bab8a714a71e795]) successfully deleted
I20260926 06:15:06.767733 4807 ts_tablet_manager.cc:1905] T 85acd211fa864ca6a16fbea04f2fb20c P 19dab98ec7184b578b218bb35dcccc85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:15:06.768977 4807 ts_tablet_manager.cc:1918] T 85acd211fa864ca6a16fbea04f2fb20c P 19dab98ec7184b578b218bb35dcccc85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 06:15:06.769043 4807 log.cc:1196] T 85acd211fa864ca6a16fbea04f2fb20c P 19dab98ec7184b578b218bb35dcccc85: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-1/wal/wals/85acd211fa864ca6a16fbea04f2fb20c
I20260926 06:15:06.769289 4807 ts_tablet_manager.cc:1939] T 85acd211fa864ca6a16fbea04f2fb20c P 19dab98ec7184b578b218bb35dcccc85: Deleting consensus metadata
I20260926 06:15:06.769603 4215 catalog_manager.cc:4841] TS 19dab98ec7184b578b218bb35dcccc85 (127.2.110.194:45967): tablet 85acd211fa864ca6a16fbea04f2fb20c (table test-table [id=2bea85a2f591465c8bab8a714a71e795]) successfully deleted
Sep 26 06:15:06 dist-test-slave-hndd krb5kdc[4185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403306, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 06:15:06 dist-test-slave-hndd krb5kdc[4185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403306, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:15:06.792032 4815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:06.783758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36830 (local address 127.2.110.254:43499)
0926 06:15:06.783918 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:06.783920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:06.784000 (+ 80us) server_negotiation.cc:407] Connection header received
0926 06:15:06.784099 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:06.784100 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:06.784113 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:06.784201 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:06.784790 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:06.785316 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:06.786158 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:06.786373 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:06.786460 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:06.789125 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:06.789127 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:06.789127 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:06.789135 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:06.790776 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:06.791160 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:06.791161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:06.791161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:06.791214 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:06.791450 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:06.791452 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:06.791452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:06.791592 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:06.791684 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:06.791870 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 06:15:06.791945 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":76,"threads_started":1}
I20260926 06:15:06.793653 4228 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36830:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:15:06.793835 4228 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:15:06.795266 4507 tablet_service.cc:1467] Processing CreateTablet for tablet 7e52efde48ae4a9ebeda76e25580a5e0 (DEFAULT_TABLE table=test-table [id=15f38fc7c52b4acfb2513cbda548d7d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:15:06.795264 4367 tablet_service.cc:1467] Processing CreateTablet for tablet 7e52efde48ae4a9ebeda76e25580a5e0 (DEFAULT_TABLE table=test-table [id=15f38fc7c52b4acfb2513cbda548d7d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:15:06.795351 4646 tablet_service.cc:1467] Processing CreateTablet for tablet 7e52efde48ae4a9ebeda76e25580a5e0 (DEFAULT_TABLE table=test-table [id=15f38fc7c52b4acfb2513cbda548d7d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:15:06.795410 4367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e52efde48ae4a9ebeda76e25580a5e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:06.795410 4507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e52efde48ae4a9ebeda76e25580a5e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:06.795461 4646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e52efde48ae4a9ebeda76e25580a5e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:06.796451 4816 tablet_bootstrap.cc:492] T 7e52efde48ae4a9ebeda76e25580a5e0 P 19dab98ec7184b578b218bb35dcccc85: Bootstrap starting.
I20260926 06:15:06.796825 4817 tablet_bootstrap.cc:492] T 7e52efde48ae4a9ebeda76e25580a5e0 P 20ab543b0b074455aae04921b83afa4e: Bootstrap starting.
I20260926 06:15:06.796941 4816 tablet_bootstrap.cc:654] T 7e52efde48ae4a9ebeda76e25580a5e0 P 19dab98ec7184b578b218bb35dcccc85: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:06.797191 4817 tablet_bootstrap.cc:654] T 7e52efde48ae4a9ebeda76e25580a5e0 P 20ab543b0b074455aae04921b83afa4e: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:06.797472 4818 tablet_bootstrap.cc:492] T 7e52efde48ae4a9ebeda76e25580a5e0 P 9736545427474bbbabc55a4fcfb264de: Bootstrap starting.
I20260926 06:15:06.797646 4816 tablet_bootstrap.cc:492] T 7e52efde48ae4a9ebeda76e25580a5e0 P 19dab98ec7184b578b218bb35dcccc85: No bootstrap required, opened a new log
I20260926 06:15:06.797698 4816 ts_tablet_manager.cc:1397] T 7e52efde48ae4a9ebeda76e25580a5e0 P 19dab98ec7184b578b218bb35dcccc85: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:06.797762 4817 tablet_bootstrap.cc:492] T 7e52efde48ae4a9ebeda76e25580a5e0 P 20ab543b0b074455aae04921b83afa4e: No bootstrap required, opened a new log
I20260926 06:15:06.797816 4817 ts_tablet_manager.cc:1397] T 7e52efde48ae4a9ebeda76e25580a5e0 P 20ab543b0b074455aae04921b83afa4e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:06.797820 4818 tablet_bootstrap.cc:654] T 7e52efde48ae4a9ebeda76e25580a5e0 P 9736545427474bbbabc55a4fcfb264de: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:06.797844 4816 raft_consensus.cc:348] T 7e52efde48ae4a9ebeda76e25580a5e0 P 19dab98ec7184b578b218bb35dcccc85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } } peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } }
I20260926 06:15:06.797959 4817 raft_consensus.cc:348] T 7e52efde48ae4a9ebeda76e25580a5e0 P 20ab543b0b074455aae04921b83afa4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } } peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } }
I20260926 06:15:06.798034 4817 raft_consensus.cc:374] T 7e52efde48ae4a9ebeda76e25580a5e0 P 20ab543b0b074455aae04921b83afa4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:06.798058 4817 raft_consensus.cc:729] T 7e52efde48ae4a9ebeda76e25580a5e0 P 20ab543b0b074455aae04921b83afa4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20ab543b0b074455aae04921b83afa4e, State: Initialized, Role: FOLLOWER
I20260926 06:15:06.798030 4816 raft_consensus.cc:374] T 7e52efde48ae4a9ebeda76e25580a5e0 P 19dab98ec7184b578b218bb35dcccc85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:06.798101 4816 raft_consensus.cc:729] T 7e52efde48ae4a9ebeda76e25580a5e0 P 19dab98ec7184b578b218bb35dcccc85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19dab98ec7184b578b218bb35dcccc85, State: Initialized, Role: FOLLOWER
I20260926 06:15:06.798122 4817 consensus_queue.cc:260] T 7e52efde48ae4a9ebeda76e25580a5e0 P 20ab543b0b074455aae04921b83afa4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } } peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } }
I20260926 06:15:06.798158 4816 consensus_queue.cc:260] T 7e52efde48ae4a9ebeda76e25580a5e0 P 19dab98ec7184b578b218bb35dcccc85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } } peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } }
I20260926 06:15:06.798316 4817 ts_tablet_manager.cc:1428] T 7e52efde48ae4a9ebeda76e25580a5e0 P 20ab543b0b074455aae04921b83afa4e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 06:15:06.798375 4816 ts_tablet_manager.cc:1428] T 7e52efde48ae4a9ebeda76e25580a5e0 P 19dab98ec7184b578b218bb35dcccc85: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:06.798441 4818 tablet_bootstrap.cc:492] T 7e52efde48ae4a9ebeda76e25580a5e0 P 9736545427474bbbabc55a4fcfb264de: No bootstrap required, opened a new log
I20260926 06:15:06.798491 4818 ts_tablet_manager.cc:1397] T 7e52efde48ae4a9ebeda76e25580a5e0 P 9736545427474bbbabc55a4fcfb264de: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:06.798950 4818 raft_consensus.cc:348] T 7e52efde48ae4a9ebeda76e25580a5e0 P 9736545427474bbbabc55a4fcfb264de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } } peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } }
I20260926 06:15:06.799054 4818 raft_consensus.cc:374] T 7e52efde48ae4a9ebeda76e25580a5e0 P 9736545427474bbbabc55a4fcfb264de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:06.799091 4818 raft_consensus.cc:729] T 7e52efde48ae4a9ebeda76e25580a5e0 P 9736545427474bbbabc55a4fcfb264de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9736545427474bbbabc55a4fcfb264de, State: Initialized, Role: FOLLOWER
I20260926 06:15:06.799139 4818 consensus_queue.cc:260] T 7e52efde48ae4a9ebeda76e25580a5e0 P 9736545427474bbbabc55a4fcfb264de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } } peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } }
I20260926 06:15:06.799238 4818 ts_tablet_manager.cc:1428] T 7e52efde48ae4a9ebeda76e25580a5e0 P 9736545427474bbbabc55a4fcfb264de: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:06.889807 4793 raft_consensus.cc:482] T 7e52efde48ae4a9ebeda76e25580a5e0 P 9736545427474bbbabc55a4fcfb264de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:15:06.889914 4793 raft_consensus.cc:504] T 7e52efde48ae4a9ebeda76e25580a5e0 P 9736545427474bbbabc55a4fcfb264de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } } peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } }
I20260926 06:15:06.890275 4793 leader_election.cc:288] T 7e52efde48ae4a9ebeda76e25580a5e0 P 9736545427474bbbabc55a4fcfb264de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20ab543b0b074455aae04921b83afa4e (127.2.110.193:46297), 19dab98ec7184b578b218bb35dcccc85 (127.2.110.194:45967)
I20260926 06:15:06.890618 4387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e52efde48ae4a9ebeda76e25580a5e0" candidate_uuid: "9736545427474bbbabc55a4fcfb264de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20ab543b0b074455aae04921b83afa4e" is_pre_election: true
I20260926 06:15:06.890720 4387 raft_consensus.cc:2454] T 7e52efde48ae4a9ebeda76e25580a5e0 P 20ab543b0b074455aae04921b83afa4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9736545427474bbbabc55a4fcfb264de in term 0.
I20260926 06:15:06.890647 4526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e52efde48ae4a9ebeda76e25580a5e0" candidate_uuid: "9736545427474bbbabc55a4fcfb264de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19dab98ec7184b578b218bb35dcccc85" is_pre_election: true
I20260926 06:15:06.890774 4526 raft_consensus.cc:2454] T 7e52efde48ae4a9ebeda76e25580a5e0 P 19dab98ec7184b578b218bb35dcccc85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9736545427474bbbabc55a4fcfb264de in term 0.
I20260926 06:15:06.890931 4600 leader_election.cc:302] T 7e52efde48ae4a9ebeda76e25580a5e0 P 9736545427474bbbabc55a4fcfb264de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20ab543b0b074455aae04921b83afa4e, 9736545427474bbbabc55a4fcfb264de; no voters:
I20260926 06:15:06.891194 4793 raft_consensus.cc:2781] T 7e52efde48ae4a9ebeda76e25580a5e0 P 9736545427474bbbabc55a4fcfb264de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:15:06.891273 4793 raft_consensus.cc:482] T 7e52efde48ae4a9ebeda76e25580a5e0 P 9736545427474bbbabc55a4fcfb264de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:15:06.891305 4793 raft_consensus.cc:3037] T 7e52efde48ae4a9ebeda76e25580a5e0 P 9736545427474bbbabc55a4fcfb264de [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:06.892745 4793 raft_consensus.cc:504] T 7e52efde48ae4a9ebeda76e25580a5e0 P 9736545427474bbbabc55a4fcfb264de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } } peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } }
I20260926 06:15:06.892920 4793 leader_election.cc:288] T 7e52efde48ae4a9ebeda76e25580a5e0 P 9736545427474bbbabc55a4fcfb264de [CANDIDATE]: Term 1 election: Requested vote from peers 20ab543b0b074455aae04921b83afa4e (127.2.110.193:46297), 19dab98ec7184b578b218bb35dcccc85 (127.2.110.194:45967)
I20260926 06:15:06.893087 4526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e52efde48ae4a9ebeda76e25580a5e0" candidate_uuid: "9736545427474bbbabc55a4fcfb264de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19dab98ec7184b578b218bb35dcccc85"
I20260926 06:15:06.893190 4526 raft_consensus.cc:3037] T 7e52efde48ae4a9ebeda76e25580a5e0 P 19dab98ec7184b578b218bb35dcccc85 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:06.893249 4387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e52efde48ae4a9ebeda76e25580a5e0" candidate_uuid: "9736545427474bbbabc55a4fcfb264de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20ab543b0b074455aae04921b83afa4e"
I20260926 06:15:06.893385 4387 raft_consensus.cc:3037] T 7e52efde48ae4a9ebeda76e25580a5e0 P 20ab543b0b074455aae04921b83afa4e [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:06.894006 4387 raft_consensus.cc:2454] T 7e52efde48ae4a9ebeda76e25580a5e0 P 20ab543b0b074455aae04921b83afa4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9736545427474bbbabc55a4fcfb264de in term 1.
I20260926 06:15:06.894285 4600 leader_election.cc:302] T 7e52efde48ae4a9ebeda76e25580a5e0 P 9736545427474bbbabc55a4fcfb264de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20ab543b0b074455aae04921b83afa4e, 9736545427474bbbabc55a4fcfb264de; no voters:
I20260926 06:15:06.894456 4793 raft_consensus.cc:2781] T 7e52efde48ae4a9ebeda76e25580a5e0 P 9736545427474bbbabc55a4fcfb264de [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:15:06.894515 4526 raft_consensus.cc:2454] T 7e52efde48ae4a9ebeda76e25580a5e0 P 19dab98ec7184b578b218bb35dcccc85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9736545427474bbbabc55a4fcfb264de in term 1.
I20260926 06:15:06.894555 4793 raft_consensus.cc:686] T 7e52efde48ae4a9ebeda76e25580a5e0 P 9736545427474bbbabc55a4fcfb264de [term 1 LEADER]: Becoming Leader. State: Replica: 9736545427474bbbabc55a4fcfb264de, State: Running, Role: LEADER
I20260926 06:15:06.894619 4793 consensus_queue.cc:237] T 7e52efde48ae4a9ebeda76e25580a5e0 P 9736545427474bbbabc55a4fcfb264de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } } peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } }
I20260926 06:15:06.895445 4228 catalog_manager.cc:5495] T 7e52efde48ae4a9ebeda76e25580a5e0 P 9736545427474bbbabc55a4fcfb264de reported cstate change: term changed from 0 to 1, leader changed from <none> to 9736545427474bbbabc55a4fcfb264de (127.2.110.195). New cstate: current_term: 1 leader_uuid: "9736545427474bbbabc55a4fcfb264de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } health_report { overall_health: HEALTHY } } }
I20260926 06:15:06.921710 4826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:06.919040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35760 (local address 127.2.110.195:36447)
0926 06:15:06.919204 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:06.919206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:06.919213 (+ 7us) server_negotiation.cc:407] Connection header received
0926 06:15:06.919232 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:06.919232 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:06.919243 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:06.919325 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:15:06.919468 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:06.920005 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:06.920850 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:06.921102 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:06.921205 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:06.921397 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:15:06.921442 (+ 45us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:15:06.921502 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:06.921576 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 06:15:06.921602 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":81,"threads_started":1}
I20260926 06:15:06.924113 4526 raft_consensus.cc:1261] T 7e52efde48ae4a9ebeda76e25580a5e0 P 19dab98ec7184b578b218bb35dcccc85 [term 1 FOLLOWER]: Refusing update from remote peer 9736545427474bbbabc55a4fcfb264de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:15:06.924149 4387 raft_consensus.cc:1261] T 7e52efde48ae4a9ebeda76e25580a5e0 P 20ab543b0b074455aae04921b83afa4e [term 1 FOLLOWER]: Refusing update from remote peer 9736545427474bbbabc55a4fcfb264de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:15:06.924384 4828 consensus_queue.cc:1035] T 7e52efde48ae4a9ebeda76e25580a5e0 P 9736545427474bbbabc55a4fcfb264de [LEADER]: Connected to new peer: Peer: permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:15:06.924517 4793 consensus_queue.cc:1035] T 7e52efde48ae4a9ebeda76e25580a5e0 P 9736545427474bbbabc55a4fcfb264de [LEADER]: Connected to new peer: Peer: permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:15:06.935233 4230 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:15:06.935381 4230 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:15:06.936102 4230 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9fdc65b166ea49feab076956712a2657: Sending DeleteTablet for 3 replicas of tablet 7e52efde48ae4a9ebeda76e25580a5e0
I20260926 06:15:06.936398 4367 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e52efde48ae4a9ebeda76e25580a5e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:15:06 UTC) from {username='kudu', principal='kudu/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:35210
I20260926 06:15:06.936496 4507 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e52efde48ae4a9ebeda76e25580a5e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:15:06 UTC) from {username='kudu', principal='kudu/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:32984
I20260926 06:15:06.936551 4805 tablet_replica.cc:330] T 7e52efde48ae4a9ebeda76e25580a5e0 P 20ab543b0b074455aae04921b83afa4e: stopping tablet replica
I20260926 06:15:06.936543 4646 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e52efde48ae4a9ebeda76e25580a5e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:15:06 UTC) from {username='kudu', principal='kudu/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:59584
I20260926 06:15:06.936591 4807 tablet_replica.cc:330] T 7e52efde48ae4a9ebeda76e25580a5e0 P 19dab98ec7184b578b218bb35dcccc85: stopping tablet replica
I20260926 06:15:06.936612 4805 raft_consensus.cc:2229] T 7e52efde48ae4a9ebeda76e25580a5e0 P 20ab543b0b074455aae04921b83afa4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:15:06.936635 4806 tablet_replica.cc:330] T 7e52efde48ae4a9ebeda76e25580a5e0 P 9736545427474bbbabc55a4fcfb264de: stopping tablet replica
I20260926 06:15:06.936651 4807 raft_consensus.cc:2229] T 7e52efde48ae4a9ebeda76e25580a5e0 P 19dab98ec7184b578b218bb35dcccc85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:15:06.936669 4805 raft_consensus.cc:2258] T 7e52efde48ae4a9ebeda76e25580a5e0 P 20ab543b0b074455aae04921b83afa4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:15:06.936676 4806 raft_consensus.cc:2229] T 7e52efde48ae4a9ebeda76e25580a5e0 P 9736545427474bbbabc55a4fcfb264de [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:15:06.936690 4807 raft_consensus.cc:2258] T 7e52efde48ae4a9ebeda76e25580a5e0 P 19dab98ec7184b578b218bb35dcccc85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:15:06.936730 4806 raft_consensus.cc:2258] T 7e52efde48ae4a9ebeda76e25580a5e0 P 9736545427474bbbabc55a4fcfb264de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:15:06.937011 4807 ts_tablet_manager.cc:1905] T 7e52efde48ae4a9ebeda76e25580a5e0 P 19dab98ec7184b578b218bb35dcccc85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:15:06.937062 4806 ts_tablet_manager.cc:1905] T 7e52efde48ae4a9ebeda76e25580a5e0 P 9736545427474bbbabc55a4fcfb264de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:15:06.937059 4805 ts_tablet_manager.cc:1905] T 7e52efde48ae4a9ebeda76e25580a5e0 P 20ab543b0b074455aae04921b83afa4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:15:06.938516 4807 ts_tablet_manager.cc:1918] T 7e52efde48ae4a9ebeda76e25580a5e0 P 19dab98ec7184b578b218bb35dcccc85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 06:15:06.938567 4807 log.cc:1196] T 7e52efde48ae4a9ebeda76e25580a5e0 P 19dab98ec7184b578b218bb35dcccc85: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-1/wal/wals/7e52efde48ae4a9ebeda76e25580a5e0
I20260926 06:15:06.938701 4806 ts_tablet_manager.cc:1918] T 7e52efde48ae4a9ebeda76e25580a5e0 P 9736545427474bbbabc55a4fcfb264de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 06:15:06.938735 4807 ts_tablet_manager.cc:1939] T 7e52efde48ae4a9ebeda76e25580a5e0 P 19dab98ec7184b578b218bb35dcccc85: Deleting consensus metadata
I20260926 06:15:06.938763 4806 log.cc:1196] T 7e52efde48ae4a9ebeda76e25580a5e0 P 9736545427474bbbabc55a4fcfb264de: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-2/wal/wals/7e52efde48ae4a9ebeda76e25580a5e0
I20260926 06:15:06.938777 4805 ts_tablet_manager.cc:1918] T 7e52efde48ae4a9ebeda76e25580a5e0 P 20ab543b0b074455aae04921b83afa4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 06:15:06.938828 4805 log.cc:1196] T 7e52efde48ae4a9ebeda76e25580a5e0 P 20ab543b0b074455aae04921b83afa4e: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-0/wal/wals/7e52efde48ae4a9ebeda76e25580a5e0
I20260926 06:15:06.939002 4806 ts_tablet_manager.cc:1939] T 7e52efde48ae4a9ebeda76e25580a5e0 P 9736545427474bbbabc55a4fcfb264de: Deleting consensus metadata
I20260926 06:15:06.939025 4215 catalog_manager.cc:4841] TS 19dab98ec7184b578b218bb35dcccc85 (127.2.110.194:45967): tablet 7e52efde48ae4a9ebeda76e25580a5e0 (table test-table [id=15f38fc7c52b4acfb2513cbda548d7d0]) successfully deleted
I20260926 06:15:06.939065 4805 ts_tablet_manager.cc:1939] T 7e52efde48ae4a9ebeda76e25580a5e0 P 20ab543b0b074455aae04921b83afa4e: Deleting consensus metadata
I20260926 06:15:06.939404 4217 catalog_manager.cc:4841] TS 9736545427474bbbabc55a4fcfb264de (127.2.110.195:36447): tablet 7e52efde48ae4a9ebeda76e25580a5e0 (table test-table [id=15f38fc7c52b4acfb2513cbda548d7d0]) successfully deleted
I20260926 06:15:06.939447 4216 catalog_manager.cc:4841] TS 20ab543b0b074455aae04921b83afa4e (127.2.110.193:46297): tablet 7e52efde48ae4a9ebeda76e25580a5e0 (table test-table [id=15f38fc7c52b4acfb2513cbda548d7d0]) successfully deleted
I20260926 06:15:06.939581 4230 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36830:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:15:06.939714 4230 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:15:06.941165 4367 tablet_service.cc:1467] Processing CreateTablet for tablet 951b4f2c1d4a4af49c03b79a4557ee3a (DEFAULT_TABLE table=test-table [id=31e361f216454fcb9247bd950ca1895c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:15:06.941236 4507 tablet_service.cc:1467] Processing CreateTablet for tablet 951b4f2c1d4a4af49c03b79a4557ee3a (DEFAULT_TABLE table=test-table [id=31e361f216454fcb9247bd950ca1895c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:15:06.941298 4367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 951b4f2c1d4a4af49c03b79a4557ee3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:06.941393 4507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 951b4f2c1d4a4af49c03b79a4557ee3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:06.942355 4817 tablet_bootstrap.cc:492] T 951b4f2c1d4a4af49c03b79a4557ee3a P 20ab543b0b074455aae04921b83afa4e: Bootstrap starting.
I20260926 06:15:06.942436 4816 tablet_bootstrap.cc:492] T 951b4f2c1d4a4af49c03b79a4557ee3a P 19dab98ec7184b578b218bb35dcccc85: Bootstrap starting.
I20260926 06:15:06.942499 4646 tablet_service.cc:1467] Processing CreateTablet for tablet 951b4f2c1d4a4af49c03b79a4557ee3a (DEFAULT_TABLE table=test-table [id=31e361f216454fcb9247bd950ca1895c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:15:06.942615 4646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 951b4f2c1d4a4af49c03b79a4557ee3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:06.942950 4816 tablet_bootstrap.cc:654] T 951b4f2c1d4a4af49c03b79a4557ee3a P 19dab98ec7184b578b218bb35dcccc85: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:06.943274 4817 tablet_bootstrap.cc:654] T 951b4f2c1d4a4af49c03b79a4557ee3a P 20ab543b0b074455aae04921b83afa4e: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:06.943693 4816 tablet_bootstrap.cc:492] T 951b4f2c1d4a4af49c03b79a4557ee3a P 19dab98ec7184b578b218bb35dcccc85: No bootstrap required, opened a new log
I20260926 06:15:06.943769 4816 ts_tablet_manager.cc:1397] T 951b4f2c1d4a4af49c03b79a4557ee3a P 19dab98ec7184b578b218bb35dcccc85: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:06.943938 4816 raft_consensus.cc:348] T 951b4f2c1d4a4af49c03b79a4557ee3a P 19dab98ec7184b578b218bb35dcccc85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } } peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } }
I20260926 06:15:06.944027 4816 raft_consensus.cc:374] T 951b4f2c1d4a4af49c03b79a4557ee3a P 19dab98ec7184b578b218bb35dcccc85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:06.944052 4816 raft_consensus.cc:729] T 951b4f2c1d4a4af49c03b79a4557ee3a P 19dab98ec7184b578b218bb35dcccc85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19dab98ec7184b578b218bb35dcccc85, State: Initialized, Role: FOLLOWER
I20260926 06:15:06.944069 4818 tablet_bootstrap.cc:492] T 951b4f2c1d4a4af49c03b79a4557ee3a P 9736545427474bbbabc55a4fcfb264de: Bootstrap starting.
I20260926 06:15:06.944108 4816 consensus_queue.cc:260] T 951b4f2c1d4a4af49c03b79a4557ee3a P 19dab98ec7184b578b218bb35dcccc85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } } peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } }
I20260926 06:15:06.944207 4816 ts_tablet_manager.cc:1428] T 951b4f2c1d4a4af49c03b79a4557ee3a P 19dab98ec7184b578b218bb35dcccc85: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 06:15:06.944386 4818 tablet_bootstrap.cc:654] T 951b4f2c1d4a4af49c03b79a4557ee3a P 9736545427474bbbabc55a4fcfb264de: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:06.944667 4817 tablet_bootstrap.cc:492] T 951b4f2c1d4a4af49c03b79a4557ee3a P 20ab543b0b074455aae04921b83afa4e: No bootstrap required, opened a new log
I20260926 06:15:06.944720 4817 ts_tablet_manager.cc:1397] T 951b4f2c1d4a4af49c03b79a4557ee3a P 20ab543b0b074455aae04921b83afa4e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 06:15:06.944880 4817 raft_consensus.cc:348] T 951b4f2c1d4a4af49c03b79a4557ee3a P 20ab543b0b074455aae04921b83afa4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } } peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } }
I20260926 06:15:06.944965 4817 raft_consensus.cc:374] T 951b4f2c1d4a4af49c03b79a4557ee3a P 20ab543b0b074455aae04921b83afa4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:06.944968 4818 tablet_bootstrap.cc:492] T 951b4f2c1d4a4af49c03b79a4557ee3a P 9736545427474bbbabc55a4fcfb264de: No bootstrap required, opened a new log
I20260926 06:15:06.944991 4817 raft_consensus.cc:729] T 951b4f2c1d4a4af49c03b79a4557ee3a P 20ab543b0b074455aae04921b83afa4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20ab543b0b074455aae04921b83afa4e, State: Initialized, Role: FOLLOWER
I20260926 06:15:06.945013 4818 ts_tablet_manager.cc:1397] T 951b4f2c1d4a4af49c03b79a4557ee3a P 9736545427474bbbabc55a4fcfb264de: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:06.945047 4817 consensus_queue.cc:260] T 951b4f2c1d4a4af49c03b79a4557ee3a P 20ab543b0b074455aae04921b83afa4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } } peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } }
I20260926 06:15:06.945158 4817 ts_tablet_manager.cc:1428] T 951b4f2c1d4a4af49c03b79a4557ee3a P 20ab543b0b074455aae04921b83afa4e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:06.945122 4818 raft_consensus.cc:348] T 951b4f2c1d4a4af49c03b79a4557ee3a P 9736545427474bbbabc55a4fcfb264de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } } peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } }
I20260926 06:15:06.945188 4818 raft_consensus.cc:374] T 951b4f2c1d4a4af49c03b79a4557ee3a P 9736545427474bbbabc55a4fcfb264de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:06.945216 4818 raft_consensus.cc:729] T 951b4f2c1d4a4af49c03b79a4557ee3a P 9736545427474bbbabc55a4fcfb264de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9736545427474bbbabc55a4fcfb264de, State: Initialized, Role: FOLLOWER
I20260926 06:15:06.945269 4818 consensus_queue.cc:260] T 951b4f2c1d4a4af49c03b79a4557ee3a P 9736545427474bbbabc55a4fcfb264de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } } peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } }
I20260926 06:15:06.945416 4818 ts_tablet_manager.cc:1428] T 951b4f2c1d4a4af49c03b79a4557ee3a P 9736545427474bbbabc55a4fcfb264de: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:07.173244 4823 raft_consensus.cc:482] T 951b4f2c1d4a4af49c03b79a4557ee3a P 19dab98ec7184b578b218bb35dcccc85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:15:07.173424 4823 raft_consensus.cc:504] T 951b4f2c1d4a4af49c03b79a4557ee3a P 19dab98ec7184b578b218bb35dcccc85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } } peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } }
I20260926 06:15:07.173851 4823 leader_election.cc:288] T 951b4f2c1d4a4af49c03b79a4557ee3a P 19dab98ec7184b578b218bb35dcccc85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20ab543b0b074455aae04921b83afa4e (127.2.110.193:46297), 9736545427474bbbabc55a4fcfb264de (127.2.110.195:36447)
I20260926 06:15:07.178246 4835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:07.174160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.193:46297 (local address 127.2.110.194:59305)
0926 06:15:07.174389 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:07.174401 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:07.174453 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:07.174801 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:07.174802 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:07.174803 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:15:07.174861 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:07.174862 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:07.175858 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:07.175859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:07.177050 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:07.177052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:07.177884 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:07.177884 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:07.177981 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:07.177986 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 06:15:07.178059 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 06:15:07.178110 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":19,"thread_start_us":146,"threads_started":1}
I20260926 06:15:07.178380 4834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:07.174069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.195:36447 (local address 127.2.110.194:52049)
0926 06:15:07.174235 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:07.174248 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:07.174325 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:07.174575 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:07.174576 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:07.174578 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:15:07.174651 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:07.174652 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:07.175712 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:07.175713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:07.177049 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:07.177052 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:07.177882 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:07.177884 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:07.177985 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:07.177987 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 06:15:07.178059 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 06:15:07.178110 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":9,"thread_start_us":75,"threads_started":1}
I20260926 06:15:07.178354 4826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:07.174167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:52049 (local address 127.2.110.195:36447)
0926 06:15:07.174247 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:07.174250 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:07.174309 (+ 59us) server_negotiation.cc:407] Connection header received
0926 06:15:07.174432 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:07.174434 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:07.174448 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:07.174486 (+ 38us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:15:07.174733 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:07.175596 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:07.177170 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:07.177780 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:07.177866 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:07.177877 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:07.178175 (+ 298us) server_negotiation.cc:299] Negotiation successful
0926 06:15:07.178255 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 06:15:07.178395 4836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:07.174309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:59305 (local address 127.2.110.193:46297)
0926 06:15:07.174612 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:07.174615 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:07.174626 (+ 11us) server_negotiation.cc:407] Connection header received
0926 06:15:07.174657 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:07.174657 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:07.174668 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:07.174789 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:15:07.174935 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:07.175738 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:07.177170 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:07.177788 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:07.177866 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:07.177931 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:07.178175 (+ 244us) server_negotiation.cc:299] Negotiation successful
0926 06:15:07.178255 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":212,"threads_started":1}
I20260926 06:15:07.178668 4387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951b4f2c1d4a4af49c03b79a4557ee3a" candidate_uuid: "19dab98ec7184b578b218bb35dcccc85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20ab543b0b074455aae04921b83afa4e" is_pre_election: true
I20260926 06:15:07.178779 4387 raft_consensus.cc:2454] T 951b4f2c1d4a4af49c03b79a4557ee3a P 20ab543b0b074455aae04921b83afa4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19dab98ec7184b578b218bb35dcccc85 in term 0.
I20260926 06:15:07.178782 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951b4f2c1d4a4af49c03b79a4557ee3a" candidate_uuid: "19dab98ec7184b578b218bb35dcccc85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9736545427474bbbabc55a4fcfb264de" is_pre_election: true
I20260926 06:15:07.178942 4667 raft_consensus.cc:2454] T 951b4f2c1d4a4af49c03b79a4557ee3a P 9736545427474bbbabc55a4fcfb264de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19dab98ec7184b578b218bb35dcccc85 in term 0.
I20260926 06:15:07.178977 4460 leader_election.cc:302] T 951b4f2c1d4a4af49c03b79a4557ee3a P 19dab98ec7184b578b218bb35dcccc85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19dab98ec7184b578b218bb35dcccc85, 20ab543b0b074455aae04921b83afa4e; no voters:
I20260926 06:15:07.179116 4823 raft_consensus.cc:2781] T 951b4f2c1d4a4af49c03b79a4557ee3a P 19dab98ec7184b578b218bb35dcccc85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:15:07.179160 4823 raft_consensus.cc:482] T 951b4f2c1d4a4af49c03b79a4557ee3a P 19dab98ec7184b578b218bb35dcccc85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:15:07.179203 4823 raft_consensus.cc:3037] T 951b4f2c1d4a4af49c03b79a4557ee3a P 19dab98ec7184b578b218bb35dcccc85 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:07.179970 4823 raft_consensus.cc:504] T 951b4f2c1d4a4af49c03b79a4557ee3a P 19dab98ec7184b578b218bb35dcccc85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } } peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } }
I20260926 06:15:07.180209 4823 leader_election.cc:288] T 951b4f2c1d4a4af49c03b79a4557ee3a P 19dab98ec7184b578b218bb35dcccc85 [CANDIDATE]: Term 1 election: Requested vote from peers 20ab543b0b074455aae04921b83afa4e (127.2.110.193:46297), 9736545427474bbbabc55a4fcfb264de (127.2.110.195:36447)
I20260926 06:15:07.180409 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951b4f2c1d4a4af49c03b79a4557ee3a" candidate_uuid: "19dab98ec7184b578b218bb35dcccc85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9736545427474bbbabc55a4fcfb264de"
I20260926 06:15:07.180454 4387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951b4f2c1d4a4af49c03b79a4557ee3a" candidate_uuid: "19dab98ec7184b578b218bb35dcccc85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20ab543b0b074455aae04921b83afa4e"
I20260926 06:15:07.180492 4667 raft_consensus.cc:3037] T 951b4f2c1d4a4af49c03b79a4557ee3a P 9736545427474bbbabc55a4fcfb264de [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:07.180521 4387 raft_consensus.cc:3037] T 951b4f2c1d4a4af49c03b79a4557ee3a P 20ab543b0b074455aae04921b83afa4e [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:07.181246 4667 raft_consensus.cc:2454] T 951b4f2c1d4a4af49c03b79a4557ee3a P 9736545427474bbbabc55a4fcfb264de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19dab98ec7184b578b218bb35dcccc85 in term 1.
I20260926 06:15:07.181246 4387 raft_consensus.cc:2454] T 951b4f2c1d4a4af49c03b79a4557ee3a P 20ab543b0b074455aae04921b83afa4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19dab98ec7184b578b218bb35dcccc85 in term 1.
I20260926 06:15:07.181443 4461 leader_election.cc:302] T 951b4f2c1d4a4af49c03b79a4557ee3a P 19dab98ec7184b578b218bb35dcccc85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19dab98ec7184b578b218bb35dcccc85, 9736545427474bbbabc55a4fcfb264de; no voters:
I20260926 06:15:07.181553 4823 raft_consensus.cc:2781] T 951b4f2c1d4a4af49c03b79a4557ee3a P 19dab98ec7184b578b218bb35dcccc85 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:15:07.181711 4823 raft_consensus.cc:686] T 951b4f2c1d4a4af49c03b79a4557ee3a P 19dab98ec7184b578b218bb35dcccc85 [term 1 LEADER]: Becoming Leader. State: Replica: 19dab98ec7184b578b218bb35dcccc85, State: Running, Role: LEADER
I20260926 06:15:07.181770 4823 consensus_queue.cc:237] T 951b4f2c1d4a4af49c03b79a4557ee3a P 19dab98ec7184b578b218bb35dcccc85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } } peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } }
I20260926 06:15:07.182512 4230 catalog_manager.cc:5495] T 951b4f2c1d4a4af49c03b79a4557ee3a P 19dab98ec7184b578b218bb35dcccc85 reported cstate change: term changed from 0 to 1, leader changed from <none> to 19dab98ec7184b578b218bb35dcccc85 (127.2.110.194). New cstate: current_term: 1 leader_uuid: "19dab98ec7184b578b218bb35dcccc85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19dab98ec7184b578b218bb35dcccc85" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 } health_report { overall_health: UNKNOWN } } }
I20260926 06:15:07.235463 4839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:07.232779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45512 (local address 127.2.110.194:45967)
0926 06:15:07.232979 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:07.232981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:07.232988 (+ 7us) server_negotiation.cc:407] Connection header received
0926 06:15:07.233023 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:07.233023 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:07.233042 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:07.233112 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:15:07.233235 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:07.233791 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:07.234495 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:07.234712 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:07.234764 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:07.235034 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:15:07.235157 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:15:07.235236 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:07.235306 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 06:15:07.235326 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":131,"threads_started":1}
I20260926 06:15:07.241060 4826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:07.238380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:60951 (local address 127.2.110.195:36447)
0926 06:15:07.238471 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:07.238473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:07.238551 (+ 78us) server_negotiation.cc:407] Connection header received
0926 06:15:07.238580 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:07.238581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:07.238594 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:07.238657 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:15:07.239053 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:07.239575 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:07.240351 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:07.240560 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:07.240627 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:07.240828 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:15:07.240857 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:15:07.240908 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:07.240983 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 06:15:07.241003 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 06:15:07.241071 4842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:07.238292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.195:36447 (local address 127.2.110.194:60951)
0926 06:15:07.238473 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:07.238483 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:07.238537 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:07.238671 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:07.238672 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:07.238673 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 06:15:07.238961 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:07.238962 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:07.239670 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:07.239671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:07.240267 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:07.240268 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:07.240662 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:07.240663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:07.240712 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:07.240726 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 06:15:07.240919 (+ 193us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 06:15:07.240921 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 06:15:07.240956 (+ 35us) client_negotiation.cc:240] Negotiation successful
0926 06:15:07.240993 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":105,"threads_started":1}
I20260926 06:15:07.243410 4387 raft_consensus.cc:1261] T 951b4f2c1d4a4af49c03b79a4557ee3a P 20ab543b0b074455aae04921b83afa4e [term 1 FOLLOWER]: Refusing update from remote peer 19dab98ec7184b578b218bb35dcccc85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:15:07.243476 4667 raft_consensus.cc:1261] T 951b4f2c1d4a4af49c03b79a4557ee3a P 9736545427474bbbabc55a4fcfb264de [term 1 FOLLOWER]: Refusing update from remote peer 19dab98ec7184b578b218bb35dcccc85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:15:07.243737 4837 consensus_queue.cc:1035] T 951b4f2c1d4a4af49c03b79a4557ee3a P 19dab98ec7184b578b218bb35dcccc85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20ab543b0b074455aae04921b83afa4e" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:15:07.243883 4823 consensus_queue.cc:1035] T 951b4f2c1d4a4af49c03b79a4557ee3a P 19dab98ec7184b578b218bb35dcccc85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9736545427474bbbabc55a4fcfb264de" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:15:07.251953 4847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:07.249289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.194:45967 (local address 127.2.110.195:43481)
0926 06:15:07.249471 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:07.249484 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:07.249550 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:07.249672 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:07.249673 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:07.249674 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 06:15:07.249723 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:07.249724 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:07.250296 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:07.250297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:07.251233 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:07.251235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:07.251618 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:07.251619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:07.251663 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:07.251668 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 06:15:07.251795 (+ 127us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 06:15:07.251796 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 06:15:07.251830 (+ 34us) client_negotiation.cc:240] Negotiation successful
0926 06:15:07.251886 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":132,"threads_started":1}
I20260926 06:15:07.251960 4839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:07.249419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:43481 (local address 127.2.110.194:45967)
0926 06:15:07.249492 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:07.249495 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:07.249536 (+ 41us) server_negotiation.cc:407] Connection header received
0926 06:15:07.249607 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:07.249608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:07.249618 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:07.249668 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:15:07.249763 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:07.250224 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:07.251323 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:07.251538 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:07.251602 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:07.251718 (+ 116us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:15:07.251744 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:15:07.251779 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:07.251862 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 06:15:07.251887 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260926 06:15:07.252532 4789 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 06:15:08.776338 4230 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:15:08.776445 4230 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:15:08.777637 4230 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9fdc65b166ea49feab076956712a2657: Sending DeleteTablet for 3 replicas of tablet 951b4f2c1d4a4af49c03b79a4557ee3a
I20260926 06:15:08.777910 4507 tablet_service.cc:1514] Processing DeleteTablet for tablet 951b4f2c1d4a4af49c03b79a4557ee3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:15:08 UTC) from {username='kudu', principal='kudu/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:32984
I20260926 06:15:08.777925 4646 tablet_service.cc:1514] Processing DeleteTablet for tablet 951b4f2c1d4a4af49c03b79a4557ee3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:15:08 UTC) from {username='kudu', principal='kudu/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:59584
I20260926 06:15:08.777923 4367 tablet_service.cc:1514] Processing DeleteTablet for tablet 951b4f2c1d4a4af49c03b79a4557ee3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:15:08 UTC) from {username='kudu', principal='kudu/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:35210
I20260926 06:15:08.778091 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 4302
I20260926 06:15:08.779080 4871 tablet_replica.cc:330] T 951b4f2c1d4a4af49c03b79a4557ee3a P 9736545427474bbbabc55a4fcfb264de: stopping tablet replica
I20260926 06:15:08.779158 4871 raft_consensus.cc:2229] T 951b4f2c1d4a4af49c03b79a4557ee3a P 9736545427474bbbabc55a4fcfb264de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:15:08.779211 4871 raft_consensus.cc:2258] T 951b4f2c1d4a4af49c03b79a4557ee3a P 9736545427474bbbabc55a4fcfb264de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:15:08.779315 4870 tablet_replica.cc:330] T 951b4f2c1d4a4af49c03b79a4557ee3a P 19dab98ec7184b578b218bb35dcccc85: stopping tablet replica
I20260926 06:15:08.779379 4870 raft_consensus.cc:2229] T 951b4f2c1d4a4af49c03b79a4557ee3a P 19dab98ec7184b578b218bb35dcccc85 [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:15:08.779474 4870 raft_consensus.cc:2258] T 951b4f2c1d4a4af49c03b79a4557ee3a P 19dab98ec7184b578b218bb35dcccc85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:15:08.779793 4870 ts_tablet_manager.cc:1905] T 951b4f2c1d4a4af49c03b79a4557ee3a P 19dab98ec7184b578b218bb35dcccc85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:15:08.781579 4871 ts_tablet_manager.cc:1905] T 951b4f2c1d4a4af49c03b79a4557ee3a P 9736545427474bbbabc55a4fcfb264de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:15:08.781792 4870 ts_tablet_manager.cc:1918] T 951b4f2c1d4a4af49c03b79a4557ee3a P 19dab98ec7184b578b218bb35dcccc85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 06:15:08.781858 4870 log.cc:1196] T 951b4f2c1d4a4af49c03b79a4557ee3a P 19dab98ec7184b578b218bb35dcccc85: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-1/wal/wals/951b4f2c1d4a4af49c03b79a4557ee3a
I20260926 06:15:08.782114 4870 ts_tablet_manager.cc:1939] T 951b4f2c1d4a4af49c03b79a4557ee3a P 19dab98ec7184b578b218bb35dcccc85: Deleting consensus metadata
I20260926 06:15:08.782644 4215 catalog_manager.cc:4841] TS 19dab98ec7184b578b218bb35dcccc85 (127.2.110.194:45967): tablet 951b4f2c1d4a4af49c03b79a4557ee3a (table test-table [id=31e361f216454fcb9247bd950ca1895c]) successfully deleted
I20260926 06:15:08.783504 4871 ts_tablet_manager.cc:1918] T 951b4f2c1d4a4af49c03b79a4557ee3a P 9736545427474bbbabc55a4fcfb264de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 06:15:08.783588 4871 log.cc:1196] T 951b4f2c1d4a4af49c03b79a4557ee3a P 9736545427474bbbabc55a4fcfb264de: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403292369990-2491-0/minicluster-data/ts-2/wal/wals/951b4f2c1d4a4af49c03b79a4557ee3a
I20260926 06:15:08.783807 4871 ts_tablet_manager.cc:1939] T 951b4f2c1d4a4af49c03b79a4557ee3a P 9736545427474bbbabc55a4fcfb264de: Deleting consensus metadata
I20260926 06:15:08.784224 4217 catalog_manager.cc:4841] TS 9736545427474bbbabc55a4fcfb264de (127.2.110.195:36447): tablet 951b4f2c1d4a4af49c03b79a4557ee3a (table test-table [id=31e361f216454fcb9247bd950ca1895c]) successfully deleted
W20260926 06:15:08.787518 4216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.110.193:46297 (error 108)
I20260926 06:15:08.788203 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 4442
I20260926 06:15:08.788239 4874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:08.787977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.193:46297 (local address 127.0.0.1:47278)
0926 06:15:08.788136 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:08.788191 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.193:46297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":70,"threads_started":1}
W20260926 06:15:08.789966 4216 catalog_manager.cc:4584] TS 20ab543b0b074455aae04921b83afa4e (127.2.110.193:46297): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 951b4f2c1d4a4af49c03b79a4557ee3a: Network error: Client connection negotiation failed: client connection to 127.2.110.193:46297: connect: Connection refused (error 111)
I20260926 06:15:08.795171 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 4582
I20260926 06:15:08.802021 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 4198
2026-09-26T06:15:08Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5930 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:15:08 dist-test-slave-hndd krb5kdc[4877](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:15:08 dist-test-slave-hndd krb5kdc[4877](info): set up 2 sockets
Sep 26 06:15:08 dist-test-slave-hndd krb5kdc[4877](info): commencing operation
krb5kdc: starting...
W20260926 06:15:09.865592 2491 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Sep 26 06:15:09 dist-test-slave-hndd krb5kdc[4877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:15:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:15:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
I20260926 06:15:09.996028 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:36405
--webserver_interface=127.2.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:35523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.110.254:36405
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:15:10.081735 4890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:10.082000 4890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:10.082041 4890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:10.083258 4890 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:15:10.083336 4890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:10.083360 4890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:10.083382 4890 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:15:10.083411 4890 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:15:10.084985 4890 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:35523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.110.254:36405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:36405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:10.085281 4890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:10.085541 4890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 06:15:10.091881 4890 server_base.cc:1034] running on GCE node
W20260926 06:15:10.091890 4895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:10.091847 4896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:10.091847 4898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:10.092376 4890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:10.092701 4890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:10.093881 4890 hybrid_clock.cc:648] HybridClock initialized: now 1790403310093869 us; error 44 us; skew 500 ppm
Sep 26 06:15:10 dist-test-slave-hndd krb5kdc[4877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403310, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:10.101899 4890 init.cc:377] Logged in from keytab as kudu/127.2.110.254@KRBTEST.COM (short username kudu)
I20260926 06:15:10.102444 4890 webserver.cc:460] Webserver started at http://127.2.110.254:46397/ using document root <none> and password file <none>
I20260926 06:15:10.102679 4890 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:10.102746 4890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:10.102900 4890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:10.103993 4890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/master-0/data/instance:
uuid: "f6b79a79488846009e49bf4435a1622c"
format_stamp: "Formatted at 2026-09-26 06:15:10 on dist-test-slave-hndd"
server_key: "feba9b463104dcb5400c09c5908f4744"
server_key_iv: "97f58e83853cfa222a76788c28c1daec"
server_key_version: "encryptionkey@0"
I20260926 06:15:10.104379 4890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/master-0/wal/instance:
uuid: "f6b79a79488846009e49bf4435a1622c"
format_stamp: "Formatted at 2026-09-26 06:15:10 on dist-test-slave-hndd"
server_key: "feba9b463104dcb5400c09c5908f4744"
server_key_iv: "97f58e83853cfa222a76788c28c1daec"
server_key_version: "encryptionkey@0"
I20260926 06:15:10.105902 4890 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:15:10.106803 4905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:10.107074 4890 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:15:10.107197 4890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "f6b79a79488846009e49bf4435a1622c"
format_stamp: "Formatted at 2026-09-26 06:15:10 on dist-test-slave-hndd"
server_key: "feba9b463104dcb5400c09c5908f4744"
server_key_iv: "97f58e83853cfa222a76788c28c1daec"
server_key_version: "encryptionkey@0"
I20260926 06:15:10.107295 4890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:10.122025 4890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:10.124378 4890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:10.124584 4890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:10.129225 4890 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.254:36405
I20260926 06:15:10.129271 4957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.254:36405 every 8 connection(s)
I20260926 06:15:10.129688 4890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:15:10.130568 4958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:10.131484 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 4890
I20260926 06:15:10.131579 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/master-0/wal/instance
I20260926 06:15:10.131762 2491 external_mini_cluster.cc:1428] Setting key d490b16c1b2ef69f6a2623efbaa56d6e
I20260926 06:15:10.133946 4958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6b79a79488846009e49bf4435a1622c: Bootstrap starting.
I20260926 06:15:10.134647 4958 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6b79a79488846009e49bf4435a1622c: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:10.134974 4958 log.cc:824] T 00000000000000000000000000000000 P f6b79a79488846009e49bf4435a1622c: Log is configured to *not* fsync() on all Append() calls
I20260926 06:15:10.135731 4958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6b79a79488846009e49bf4435a1622c: No bootstrap required, opened a new log
I20260926 06:15:10.138832 4958 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6b79a79488846009e49bf4435a1622c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6b79a79488846009e49bf4435a1622c" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 36405 } }
I20260926 06:15:10.139031 4958 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6b79a79488846009e49bf4435a1622c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:10.139075 4958 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6b79a79488846009e49bf4435a1622c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b79a79488846009e49bf4435a1622c, State: Initialized, Role: FOLLOWER
I20260926 06:15:10.139246 4958 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6b79a79488846009e49bf4435a1622c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6b79a79488846009e49bf4435a1622c" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 36405 } }
I20260926 06:15:10.139367 4958 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6b79a79488846009e49bf4435a1622c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:15:10.139416 4958 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6b79a79488846009e49bf4435a1622c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:15:10.139470 4958 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6b79a79488846009e49bf4435a1622c [term 0 FOLLOWER]: Advancing to term 1
Sep 26 06:15:10 dist-test-slave-hndd krb5kdc[4877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:15:10.140190 4958 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6b79a79488846009e49bf4435a1622c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6b79a79488846009e49bf4435a1622c" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 36405 } }
I20260926 06:15:10.140326 4958 leader_election.cc:302] T 00000000000000000000000000000000 P f6b79a79488846009e49bf4435a1622c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6b79a79488846009e49bf4435a1622c; no voters:
I20260926 06:15:10.140576 4958 leader_election.cc:288] T 00000000000000000000000000000000 P f6b79a79488846009e49bf4435a1622c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:15:10.140614 4963 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6b79a79488846009e49bf4435a1622c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:15:10.140789 4963 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6b79a79488846009e49bf4435a1622c [term 1 LEADER]: Becoming Leader. State: Replica: f6b79a79488846009e49bf4435a1622c, State: Running, Role: LEADER
I20260926 06:15:10.140882 4958 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6b79a79488846009e49bf4435a1622c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:15:10.140877 4963 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6b79a79488846009e49bf4435a1622c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6b79a79488846009e49bf4435a1622c" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 36405 } }
I20260926 06:15:10.141250 4963 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6b79a79488846009e49bf4435a1622c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6b79a79488846009e49bf4435a1622c. Latest consensus state: current_term: 1 leader_uuid: "f6b79a79488846009e49bf4435a1622c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6b79a79488846009e49bf4435a1622c" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 36405 } } }
I20260926 06:15:10.141368 4963 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6b79a79488846009e49bf4435a1622c [sys.catalog]: This master's current role is: LEADER
I20260926 06:15:10.141507 4964 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6b79a79488846009e49bf4435a1622c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6b79a79488846009e49bf4435a1622c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6b79a79488846009e49bf4435a1622c" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 36405 } } }
I20260926 06:15:10.141592 4969 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:15:10.141598 4964 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6b79a79488846009e49bf4435a1622c [sys.catalog]: This master's current role is: LEADER
I20260926 06:15:10.142185 4969 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:15:10.144260 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:10.132324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56004 (local address 127.2.110.254:36405)
0926 06:15:10.132518 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:10.132526 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:10.132553 (+ 27us) server_negotiation.cc:407] Connection header received
0926 06:15:10.133114 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:10.133129 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:10.133234 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:10.133548 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:10.134382 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:10.135457 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:10.136761 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:10.137213 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:10.137307 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:10.140291 (+ 2984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:10.140295 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:10.140295 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:10.140311 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:10.142496 (+ 2185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:10.142897 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:10.142900 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:10.142900 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:10.143004 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:10.143232 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:10.143233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:10.143234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:10.143573 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:10.143751 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:10.143882 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 06:15:10.144044 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":102,"threads_started":1}
I20260926 06:15:10.144521 4969 catalog_manager.cc:1347] Generated new cluster ID: 9ab54f9905324566835c95622426291e
I20260926 06:15:10.144832 4969 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:15:10.154290 4969 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:15:10.154836 4969 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:15:10.159490 4969 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6b79a79488846009e49bf4435a1622c: Generated new TSK 0
I20260926 06:15:10.159705 4969 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab.
Entry for principal kudu/127.2.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab.
I20260926 06:15:10.196918 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.193:0
--local_ip_for_outbound_sockets=127.2.110.193
--webserver_interface=127.2.110.193
--webserver_port=0
--tserver_master_addrs=127.2.110.254:36405
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab
--principal=kudu/127.2.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:35523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:15:10.291630 4984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:10.291862 4984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:10.291888 4984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:10.293252 4984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:10.293304 4984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:10.293386 4984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.193
I20260926 06:15:10.294960 4984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:35523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab
--principal=kudu/127.2.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.110.193
--webserver_port=0
--tserver_master_addrs=127.2.110.254:36405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.193
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:10.295255 4984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:10.295507 4984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:10.300863 4992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:10.300858 4990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:10.300860 4989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:10.301486 4984 server_base.cc:1034] running on GCE node
I20260926 06:15:10.301769 4984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:10.302055 4984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:10.303234 4984 hybrid_clock.cc:648] HybridClock initialized: now 1790403310303244 us; error 75 us; skew 500 ppm
Sep 26 06:15:10 dist-test-slave-hndd krb5kdc[4877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403310, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:10.306466 4984 init.cc:377] Logged in from keytab as kudu/127.2.110.193@KRBTEST.COM (short username kudu)
I20260926 06:15:10.306929 4984 webserver.cc:460] Webserver started at http://127.2.110.193:40757/ using document root <none> and password file <none>
I20260926 06:15:10.307353 4984 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:10.307454 4984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:10.307626 4984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:10.309083 4984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-0/data/instance:
uuid: "60e057e9ca70465baaf5222424263a54"
format_stamp: "Formatted at 2026-09-26 06:15:10 on dist-test-slave-hndd"
server_key: "abe547382288f2a2a73d7995d67aea5d"
server_key_iv: "20c6a5e952072763611b420b9aaca90b"
server_key_version: "encryptionkey@0"
I20260926 06:15:10.309592 4984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-0/wal/instance:
uuid: "60e057e9ca70465baaf5222424263a54"
format_stamp: "Formatted at 2026-09-26 06:15:10 on dist-test-slave-hndd"
server_key: "abe547382288f2a2a73d7995d67aea5d"
server_key_iv: "20c6a5e952072763611b420b9aaca90b"
server_key_version: "encryptionkey@0"
I20260926 06:15:10.311930 4984 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.002s
I20260926 06:15:10.313133 4999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:10.313455 4984 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:10.313583 4984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-0/wal
uuid: "60e057e9ca70465baaf5222424263a54"
format_stamp: "Formatted at 2026-09-26 06:15:10 on dist-test-slave-hndd"
server_key: "abe547382288f2a2a73d7995d67aea5d"
server_key_iv: "20c6a5e952072763611b420b9aaca90b"
server_key_version: "encryptionkey@0"
I20260926 06:15:10.313680 4984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:10.332777 4984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:10.335264 4984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:10.335435 4984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:10.335685 4984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:15:10.336014 4984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:15:10.336069 4984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:10.336117 4984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:15:10.336153 4984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:10.342682 4984 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.193:35147
I20260926 06:15:10.342811 5112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.193:35147 every 8 connection(s)
I20260926 06:15:10.343112 4984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
I20260926 06:15:10.343366 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 4984
I20260926 06:15:10.343475 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-0/wal/instance
I20260926 06:15:10.343681 2491 external_mini_cluster.cc:1428] Setting key 81cf6d1208a2d8888d1753bffc50c077
Sep 26 06:15:10 dist-test-slave-hndd krb5kdc[4877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403310, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.193@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.110.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.194@KRBTEST.COM" created.
I20260926 06:15:10.356119 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:10.344098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:32955 (local address 127.2.110.254:36405)
0926 06:15:10.344149 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:10.344151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:10.344448 (+ 297us) server_negotiation.cc:407] Connection header received
0926 06:15:10.345225 (+ 777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:10.345226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:10.345237 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:10.345293 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:10.346520 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:10.347275 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:10.348675 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:10.349064 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:10.349160 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:10.352104 (+ 2944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:10.352107 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:10.352108 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:10.352119 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:10.353908 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:10.354767 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:10.354768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:10.354769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:10.354840 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:10.355325 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:10.355326 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:10.355327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:10.355476 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:10.355560 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:10.355852 (+ 292us) server_negotiation.cc:299] Negotiation successful
0926 06:15:10.355972 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 06:15:10.356150 5115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:10.344126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:36405 (local address 127.2.110.193:32955)
0926 06:15:10.344321 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:10.344366 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:10.345102 (+ 736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:10.345396 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:10.345400 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:10.346202 (+ 802us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:10.346406 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:10.346412 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:10.347502 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:10.347504 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:10.348534 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:10.348538 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:10.349204 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:10.349206 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:10.349419 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:10.350116 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:10.350125 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:10.351955 (+ 1830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:10.354130 (+ 2175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:10.354146 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:10.354150 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:10.354574 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:10.355015 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:10.355017 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:10.355017 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:10.355239 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:10.355559 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:10.355566 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:10.355710 (+ 144us) client_negotiation.cc:769] Sending connection context
0926 06:15:10.355807 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 06:15:10.355962 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":114,"threads_started":1}
I20260926 06:15:10.356766 5113 heartbeater.cc:347] Connected to a master server at 127.2.110.254:36405
I20260926 06:15:10.356897 5113 heartbeater.cc:464] Registering TS with master...
I20260926 06:15:10.357133 5113 heartbeater.cc:511] Master 127.2.110.254:36405 requested a full tablet report, sending...
I20260926 06:15:10.357734 4922 ts_manager.cc:194] Registered new tserver with Master: 60e057e9ca70465baaf5222424263a54 (127.2.110.193:35147)
I20260926 06:15:10.358657 4922 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.193@KRBTEST.COM'} at 127.2.110.193:32955
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab.
Entry for principal kudu/127.2.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab.
I20260926 06:15:10.372023 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.194:0
--local_ip_for_outbound_sockets=127.2.110.194
--webserver_interface=127.2.110.194
--webserver_port=0
--tserver_master_addrs=127.2.110.254:36405
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab
--principal=kudu/127.2.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:35523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:15:10.458629 5118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:10.458856 5118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:10.458882 5118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:10.460073 5118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:10.460119 5118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:10.460147 5118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.194
I20260926 06:15:10.461668 5118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:35523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab
--principal=kudu/127.2.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.110.194
--webserver_port=0
--tserver_master_addrs=127.2.110.254:36405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.194
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:10.461874 5118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:10.462061 5118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:10.467070 5124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:10.467079 5126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:10.467087 5123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:10.467301 5118 server_base.cc:1034] running on GCE node
I20260926 06:15:10.467468 5118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:10.467708 5118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:10.468860 5118 hybrid_clock.cc:648] HybridClock initialized: now 1790403310468844 us; error 32 us; skew 500 ppm
Sep 26 06:15:10 dist-test-slave-hndd krb5kdc[4877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403310, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:10.471545 5118 init.cc:377] Logged in from keytab as kudu/127.2.110.194@KRBTEST.COM (short username kudu)
I20260926 06:15:10.472023 5118 webserver.cc:460] Webserver started at http://127.2.110.194:34257/ using document root <none> and password file <none>
I20260926 06:15:10.472270 5118 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:10.472339 5118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:10.472498 5118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:10.473707 5118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-1/data/instance:
uuid: "7902e05fcb724bd7a15f7c757e4d513e"
format_stamp: "Formatted at 2026-09-26 06:15:10 on dist-test-slave-hndd"
server_key: "427839d6b6e5550f5e14cc29848810bb"
server_key_iv: "2c952ef3cc34f11b7353882660545986"
server_key_version: "encryptionkey@0"
I20260926 06:15:10.474074 5118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-1/wal/instance:
uuid: "7902e05fcb724bd7a15f7c757e4d513e"
format_stamp: "Formatted at 2026-09-26 06:15:10 on dist-test-slave-hndd"
server_key: "427839d6b6e5550f5e14cc29848810bb"
server_key_iv: "2c952ef3cc34f11b7353882660545986"
server_key_version: "encryptionkey@0"
I20260926 06:15:10.475419 5118 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:10.476207 5133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:10.476382 5118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:10.476490 5118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-1/wal
uuid: "7902e05fcb724bd7a15f7c757e4d513e"
format_stamp: "Formatted at 2026-09-26 06:15:10 on dist-test-slave-hndd"
server_key: "427839d6b6e5550f5e14cc29848810bb"
server_key_iv: "2c952ef3cc34f11b7353882660545986"
server_key_version: "encryptionkey@0"
I20260926 06:15:10.476567 5118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:10.496668 5118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:10.499432 5118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:10.499603 5118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:10.499889 5118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:15:10.500226 5118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:15:10.500276 5118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:10.500303 5118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:15:10.500316 5118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:10.506793 5118 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.194:38017
I20260926 06:15:10.506886 5246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.194:38017 every 8 connection(s)
I20260926 06:15:10.507236 5118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
I20260926 06:15:10.507534 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 5118
I20260926 06:15:10.507645 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-1/wal/instance
I20260926 06:15:10.507867 2491 external_mini_cluster.cc:1428] Setting key 685213fc9ccf7f25743ee603aea23a91
Sep 26 06:15:10 dist-test-slave-hndd krb5kdc[4877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403310, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.194@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.110.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.195@KRBTEST.COM" created.
I20260926 06:15:10.519827 5249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:10.508207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:36405 (local address 127.2.110.194:56791)
0926 06:15:10.508375 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:10.508423 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:10.508995 (+ 572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:10.509251 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:10.509254 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:10.510010 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:10.510360 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:10.510376 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:10.511381 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:10.511382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:10.512321 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:10.512323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:10.512872 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:10.512874 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:10.513170 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:10.514066 (+ 896us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:10.514077 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:10.515751 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:10.518073 (+ 2322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:10.518085 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:10.518088 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:10.518524 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:10.518740 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:10.518741 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:10.518741 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:10.518851 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:10.519194 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:10.519201 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:10.519364 (+ 163us) client_negotiation.cc:769] Sending connection context
0926 06:15:10.519478 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 06:15:10.519635 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":94,"threads_started":1}
I20260926 06:15:10.520565 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:10.508276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:56791 (local address 127.2.110.254:36405)
0926 06:15:10.508350 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:10.508352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:10.508514 (+ 162us) server_negotiation.cc:407] Connection header received
0926 06:15:10.509101 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:10.509102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:10.509114 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:10.509192 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:10.510498 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:10.511275 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:10.512395 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:10.512751 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:10.512836 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:10.515894 (+ 3058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:10.515896 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:10.515897 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:10.515907 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:10.517946 (+ 2039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:10.518618 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:10.518619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:10.518619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:10.518675 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:10.518932 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:10.518933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:10.518933 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:10.519106 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:10.519183 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:10.520311 (+ 1128us) server_negotiation.cc:299] Negotiation successful
0926 06:15:10.520406 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 06:15:10.521159 5247 heartbeater.cc:347] Connected to a master server at 127.2.110.254:36405
I20260926 06:15:10.521443 5247 heartbeater.cc:464] Registering TS with master...
I20260926 06:15:10.521740 5247 heartbeater.cc:511] Master 127.2.110.254:36405 requested a full tablet report, sending...
I20260926 06:15:10.522365 4922 ts_manager.cc:194] Registered new tserver with Master: 7902e05fcb724bd7a15f7c757e4d513e (127.2.110.194:38017)
I20260926 06:15:10.522930 4922 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.194@KRBTEST.COM'} at 127.2.110.194:56791
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab.
Entry for principal kudu/127.2.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab.
I20260926 06:15:10.536577 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.195:0
--local_ip_for_outbound_sockets=127.2.110.195
--webserver_interface=127.2.110.195
--webserver_port=0
--tserver_master_addrs=127.2.110.254:36405
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab
--principal=kudu/127.2.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:35523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:15:10.626417 5252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:10.626657 5252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:10.626683 5252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:10.627925 5252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:10.627971 5252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:10.628000 5252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.195
I20260926 06:15:10.629582 5252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:35523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab
--principal=kudu/127.2.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.110.195
--webserver_port=0
--tserver_master_addrs=127.2.110.254:36405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.195
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:10.629812 5252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:10.630000 5252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:10.635664 5257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:10.635656 5258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:10.635757 5252 server_base.cc:1034] running on GCE node
W20260926 06:15:10.635679 5260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:10.636135 5252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:10.636423 5252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:10.637576 5252 hybrid_clock.cc:648] HybridClock initialized: now 1790403310637559 us; error 34 us; skew 500 ppm
Sep 26 06:15:10 dist-test-slave-hndd krb5kdc[4877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403310, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:10.641523 5252 init.cc:377] Logged in from keytab as kudu/127.2.110.195@KRBTEST.COM (short username kudu)
I20260926 06:15:10.642143 5252 webserver.cc:460] Webserver started at http://127.2.110.195:34697/ using document root <none> and password file <none>
I20260926 06:15:10.642417 5252 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:10.642500 5252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:10.642676 5252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:10.644270 5252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-2/data/instance:
uuid: "a5b29c6294014dfcbbac139cdd5417a9"
format_stamp: "Formatted at 2026-09-26 06:15:10 on dist-test-slave-hndd"
server_key: "bd375f61ba81e05fe879a2b768155265"
server_key_iv: "27fe4d9afda434c59228b851ff8e16e9"
server_key_version: "encryptionkey@0"
I20260926 06:15:10.644704 5252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-2/wal/instance:
uuid: "a5b29c6294014dfcbbac139cdd5417a9"
format_stamp: "Formatted at 2026-09-26 06:15:10 on dist-test-slave-hndd"
server_key: "bd375f61ba81e05fe879a2b768155265"
server_key_iv: "27fe4d9afda434c59228b851ff8e16e9"
server_key_version: "encryptionkey@0"
I20260926 06:15:10.646667 5252 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260926 06:15:10.647637 5267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:10.647863 5252 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:10.647943 5252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-2/wal
uuid: "a5b29c6294014dfcbbac139cdd5417a9"
format_stamp: "Formatted at 2026-09-26 06:15:10 on dist-test-slave-hndd"
server_key: "bd375f61ba81e05fe879a2b768155265"
server_key_iv: "27fe4d9afda434c59228b851ff8e16e9"
server_key_version: "encryptionkey@0"
I20260926 06:15:10.648046 5252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:10.664186 5252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:10.667249 5252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:10.667438 5252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:10.667724 5252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:15:10.668074 5252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:15:10.668135 5252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:10.668186 5252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:15:10.668232 5252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:10.675045 5252 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.195:37537
I20260926 06:15:10.675211 5380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.195:37537 every 8 connection(s)
I20260926 06:15:10.675504 5252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
I20260926 06:15:10.682368 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 5252
I20260926 06:15:10.682479 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-2/wal/instance
I20260926 06:15:10.682690 2491 external_mini_cluster.cc:1428] Setting key 971d754b90abca75c253889d423f784f
Sep 26 06:15:10 dist-test-slave-hndd krb5kdc[4877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403310, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.195@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:15:10.687242 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:10.676402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:34101 (local address 127.2.110.254:36405)
0926 06:15:10.676569 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:10.676573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:10.676926 (+ 353us) server_negotiation.cc:407] Connection header received
0926 06:15:10.677622 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:10.677623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:10.677638 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:10.677712 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:10.678747 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:10.679269 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:10.680386 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:10.680594 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:10.680692 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:10.683576 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:10.683577 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:10.683577 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:10.683584 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:10.685231 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:10.685771 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:10.685773 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:10.685773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:10.685849 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:10.686251 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:10.686252 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:10.686253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:10.686381 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:10.686443 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:10.687087 (+ 644us) server_negotiation.cc:299] Negotiation successful
0926 06:15:10.687170 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260926 06:15:10.687630 5383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:10.676303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:36405 (local address 127.2.110.195:34101)
0926 06:15:10.676612 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:10.676681 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:10.677494 (+ 813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:10.677794 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:10.677799 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:10.678421 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:10.678633 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:10.678638 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:10.679382 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:10.679383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:10.680287 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:10.680290 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:10.680702 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:10.680703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:10.680843 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:10.681478 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:10.681487 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:10.683444 (+ 1957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:10.685389 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:10.685398 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:10.685407 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:10.685656 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:10.686019 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:10.686019 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:10.686020 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:10.686134 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:10.686448 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:10.686452 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:10.686786 (+ 334us) client_negotiation.cc:769] Sending connection context
0926 06:15:10.687005 (+ 219us) client_negotiation.cc:240] Negotiation successful
0926 06:15:10.687226 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":96,"threads_started":1}
I20260926 06:15:10.688424 5381 heartbeater.cc:347] Connected to a master server at 127.2.110.254:36405
I20260926 06:15:10.688542 5381 heartbeater.cc:464] Registering TS with master...
I20260926 06:15:10.688712 5381 heartbeater.cc:511] Master 127.2.110.254:36405 requested a full tablet report, sending...
I20260926 06:15:10.689234 4922 ts_manager.cc:194] Registered new tserver with Master: a5b29c6294014dfcbbac139cdd5417a9 (127.2.110.195:37537)
I20260926 06:15:10.689707 4922 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.195@KRBTEST.COM'} at 127.2.110.195:34101
I20260926 06:15:10.694964 2491 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 06:15:10.707561 4961 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:10.706435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56006 (local address 127.2.110.254:36405)
0926 06:15:10.706516 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:10.706519 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:10.706681 (+ 162us) server_negotiation.cc:407] Connection header received
0926 06:15:10.706729 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:10.706730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:10.706742 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:10.706808 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:10.707469 (+ 661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56006: BlockingRecv error: recv got EOF from 127.0.0.1:56006 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 06:15:10.708419 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 4984
I20260926 06:15:10.713143 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 5118
I20260926 06:15:10.717453 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 5252
I20260926 06:15:10.721890 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 4890
2026-09-26T06:15:10Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1918 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790403292369990-2491-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:15:10 dist-test-slave-hndd krb5kdc[5393](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:15:10 dist-test-slave-hndd krb5kdc[5393](info): set up 2 sockets
Sep 26 06:15:10 dist-test-slave-hndd krb5kdc[5393](info): commencing operation
krb5kdc: starting...
W20260926 06:15:11.789808 2491 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Sep 26 06:15:11 dist-test-slave-hndd krb5kdc[5393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:15:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:15:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
I20260926 06:15:11.912598 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:40597
--webserver_interface=127.2.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:44779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.110.254:40597
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:15:11.993728 5406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:11.993964 5406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:11.993990 5406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:11.995254 5406 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:15:11.995306 5406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:11.995321 5406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:11.995337 5406 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:15:11.995348 5406 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:15:11.996985 5406 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:44779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790403292369990-2491-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.110.254:40597
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:40597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:11.997287 5406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:11.997555 5406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:12.002578 5412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:12.002560 5411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:12.002560 5414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:12.002831 5406 server_base.cc:1034] running on GCE node
I20260926 06:15:12.003144 5406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:12.003443 5406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:12.004594 5406 hybrid_clock.cc:648] HybridClock initialized: now 1790403312004570 us; error 38 us; skew 500 ppm
Sep 26 06:15:12 dist-test-slave-hndd krb5kdc[5393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403312, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:12.007543 5406 init.cc:377] Logged in from keytab as kudu/127.2.110.254@KRBTEST.COM (short username kudu)
I20260926 06:15:12.007966 5406 webserver.cc:460] Webserver started at http://127.2.110.254:38587/ using document root <none> and password file <none>
I20260926 06:15:12.008215 5406 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:12.008262 5406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:12.008425 5406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:12.009639 5406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790403292369990-2491-0/minicluster-data/master-0/data/instance:
uuid: "643412a51782416692571d5b85b703d3"
format_stamp: "Formatted at 2026-09-26 06:15:12 on dist-test-slave-hndd"
server_key: "727eb6c788c6b06fddfca519cfaa8e30"
server_key_iv: "8362ff0fac3ef7c32e0bc5736d2a029b"
server_key_version: "encryptionkey@0"
I20260926 06:15:12.010010 5406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790403292369990-2491-0/minicluster-data/master-0/wal/instance:
uuid: "643412a51782416692571d5b85b703d3"
format_stamp: "Formatted at 2026-09-26 06:15:12 on dist-test-slave-hndd"
server_key: "727eb6c788c6b06fddfca519cfaa8e30"
server_key_iv: "8362ff0fac3ef7c32e0bc5736d2a029b"
server_key_version: "encryptionkey@0"
I20260926 06:15:12.011404 5406 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:15:12.012193 5421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:12.012370 5406 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:12.012490 5406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "643412a51782416692571d5b85b703d3"
format_stamp: "Formatted at 2026-09-26 06:15:12 on dist-test-slave-hndd"
server_key: "727eb6c788c6b06fddfca519cfaa8e30"
server_key_iv: "8362ff0fac3ef7c32e0bc5736d2a029b"
server_key_version: "encryptionkey@0"
I20260926 06:15:12.012585 5406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:12.032665 5406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:12.035185 5406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:12.035368 5406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:12.039674 5406 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.254:40597
I20260926 06:15:12.039710 5473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.254:40597 every 8 connection(s)
I20260926 06:15:12.040073 5406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:15:12.041052 5474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:12.043453 5474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 643412a51782416692571d5b85b703d3: Bootstrap starting.
I20260926 06:15:12.044071 5474 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 643412a51782416692571d5b85b703d3: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:12.044312 5474 log.cc:824] T 00000000000000000000000000000000 P 643412a51782416692571d5b85b703d3: Log is configured to *not* fsync() on all Append() calls
I20260926 06:15:12.045046 5474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 643412a51782416692571d5b85b703d3: No bootstrap required, opened a new log
I20260926 06:15:12.047554 5474 raft_consensus.cc:348] T 00000000000000000000000000000000 P 643412a51782416692571d5b85b703d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "643412a51782416692571d5b85b703d3" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 40597 } }
I20260926 06:15:12.047816 5474 raft_consensus.cc:374] T 00000000000000000000000000000000 P 643412a51782416692571d5b85b703d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:12.047874 5474 raft_consensus.cc:729] T 00000000000000000000000000000000 P 643412a51782416692571d5b85b703d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 643412a51782416692571d5b85b703d3, State: Initialized, Role: FOLLOWER
I20260926 06:15:12.048095 5474 consensus_queue.cc:260] T 00000000000000000000000000000000 P 643412a51782416692571d5b85b703d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "643412a51782416692571d5b85b703d3" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 40597 } }
I20260926 06:15:12.048134 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 5406
I20260926 06:15:12.048200 5474 raft_consensus.cc:388] T 00000000000000000000000000000000 P 643412a51782416692571d5b85b703d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:15:12.048228 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790403292369990-2491-0/minicluster-data/master-0/wal/instance
I20260926 06:15:12.048246 5474 raft_consensus.cc:482] T 00000000000000000000000000000000 P 643412a51782416692571d5b85b703d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:15:12.048297 5474 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 643412a51782416692571d5b85b703d3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:12.048462 2491 external_mini_cluster.cc:1428] Setting key 58549ceda2ec9a45f7d68f33e580a41a
I20260926 06:15:12.049768 5474 raft_consensus.cc:504] T 00000000000000000000000000000000 P 643412a51782416692571d5b85b703d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "643412a51782416692571d5b85b703d3" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 40597 } }
I20260926 06:15:12.049938 5474 leader_election.cc:302] T 00000000000000000000000000000000 P 643412a51782416692571d5b85b703d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 643412a51782416692571d5b85b703d3; no voters:
I20260926 06:15:12.050143 5474 leader_election.cc:288] T 00000000000000000000000000000000 P 643412a51782416692571d5b85b703d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:15:12.050205 5479 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 643412a51782416692571d5b85b703d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:15:12.050599 5474 sys_catalog.cc:564] T 00000000000000000000000000000000 P 643412a51782416692571d5b85b703d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:15:12.050735 5479 raft_consensus.cc:686] T 00000000000000000000000000000000 P 643412a51782416692571d5b85b703d3 [term 1 LEADER]: Becoming Leader. State: Replica: 643412a51782416692571d5b85b703d3, State: Running, Role: LEADER
I20260926 06:15:12.050860 5479 consensus_queue.cc:237] T 00000000000000000000000000000000 P 643412a51782416692571d5b85b703d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "643412a51782416692571d5b85b703d3" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 40597 } }
I20260926 06:15:12.051259 5479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 643412a51782416692571d5b85b703d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 643412a51782416692571d5b85b703d3. Latest consensus state: current_term: 1 leader_uuid: "643412a51782416692571d5b85b703d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "643412a51782416692571d5b85b703d3" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 40597 } } }
I20260926 06:15:12.051499 5479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 643412a51782416692571d5b85b703d3 [sys.catalog]: This master's current role is: LEADER
I20260926 06:15:12.051563 5480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 643412a51782416692571d5b85b703d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "643412a51782416692571d5b85b703d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "643412a51782416692571d5b85b703d3" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 40597 } } }
I20260926 06:15:12.051638 5480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 643412a51782416692571d5b85b703d3 [sys.catalog]: This master's current role is: LEADER
I20260926 06:15:12.051851 5486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:15:12.052425 5486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:15:12.053953 5496 mvcc.cc:205] Tried to move back new op lower bound from 7333491966171168768 to 7333491966160764928. Current Snapshot: MvccSnapshot[applied={T|T < 7333491966171168768}]
I20260926 06:15:12.054502 5486 catalog_manager.cc:1347] Generated new cluster ID: ba62155ab5074fb0a9fb561a16e923cb
I20260926 06:15:12.054565 5486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 06:15:12 dist-test-slave-hndd krb5kdc[5393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:15:12.062278 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:12.049055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43270 (local address 127.2.110.254:40597)
0926 06:15:12.049440 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:12.049452 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:12.049500 (+ 48us) server_negotiation.cc:407] Connection header received
0926 06:15:12.049629 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:12.049635 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:12.049688 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:12.049855 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:12.051087 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:12.053043 (+ 1956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:12.054201 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:12.054764 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:12.055033 (+ 269us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:12.057817 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:12.057822 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:12.057822 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:12.057838 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:12.060026 (+ 2188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:12.060464 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:12.060467 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:12.060468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:12.060607 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:12.060823 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:12.060824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:12.060824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:12.061167 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:12.061327 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:12.061883 (+ 556us) server_negotiation.cc:299] Negotiation successful
0926 06:15:12.062036 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":163,"threads_started":1}
I20260926 06:15:12.063480 2491 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:15:12.066599 5486 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:15:12.067440 5486 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:15:12.071143 5486 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 643412a51782416692571d5b85b703d3: Generated new TSK 0
I20260926 06:15:12.071380 5486 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:15:12.083710 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:12.079303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43282 (local address 127.2.110.254:40597)
0926 06:15:12.079360 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:12.079363 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:12.079469 (+ 106us) server_negotiation.cc:407] Connection header received
0926 06:15:12.079555 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:12.079557 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:12.079570 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:12.079636 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:12.080009 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:12.080909 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:12.081818 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:12.082218 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:12.082304 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:12.083114 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:12.083116 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:12.083116 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:15:12.083400 (+ 284us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:12.083421 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:15:12.083430 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:12.083517 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:12.083574 (+ 57us) server_negotiation.cc:299] Negotiation successful
0926 06:15:12.083608 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 06:15:12.087180 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 5406
2026-09-26T06:15:12Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1362 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:15:12 dist-test-slave-hndd krb5kdc[5507](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:15:12 dist-test-slave-hndd krb5kdc[5507](info): set up 2 sockets
Sep 26 06:15:12 dist-test-slave-hndd krb5kdc[5507](info): commencing operation
krb5kdc: starting...
W20260926 06:15:13.137988 2491 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Sep 26 06:15:13 dist-test-slave-hndd krb5kdc[5507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:15:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:15:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
I20260926 06:15:13.272626 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:35407
--webserver_interface=127.2.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:33899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.110.254:35407
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:15:13.359448 5520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:13.359683 5520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:13.359706 5520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:13.361054 5520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:15:13.361119 5520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:13.361135 5520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:13.361150 5520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:15:13.361162 5520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:15:13.362866 5520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:33899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.110.254:35407
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:35407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:13.363099 5520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:13.363291 5520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:13.368583 5526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:13.368588 5525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:13.368692 5528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:13.369308 5520 server_base.cc:1034] running on GCE node
I20260926 06:15:13.369727 5520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:13.370052 5520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:13.371220 5520 hybrid_clock.cc:648] HybridClock initialized: now 1790403313371197 us; error 43 us; skew 500 ppm
Sep 26 06:15:13 dist-test-slave-hndd krb5kdc[5507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403313, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:13.374079 5520 init.cc:377] Logged in from keytab as kudu/127.2.110.254@KRBTEST.COM (short username kudu)
I20260926 06:15:13.374588 5520 webserver.cc:460] Webserver started at http://127.2.110.254:39019/ using document root <none> and password file <none>
I20260926 06:15:13.374805 5520 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:13.374847 5520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:13.375065 5520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:13.376209 5520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/master-0/data/instance:
uuid: "3de72bcd61ce487cb81c681a1726de74"
format_stamp: "Formatted at 2026-09-26 06:15:13 on dist-test-slave-hndd"
server_key: "f6a3846dc6537f650fe4a1451fc021dd"
server_key_iv: "1cfdd39fad3547fe5ae0b58617924c8a"
server_key_version: "encryptionkey@0"
I20260926 06:15:13.376520 5520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/master-0/wal/instance:
uuid: "3de72bcd61ce487cb81c681a1726de74"
format_stamp: "Formatted at 2026-09-26 06:15:13 on dist-test-slave-hndd"
server_key: "f6a3846dc6537f650fe4a1451fc021dd"
server_key_iv: "1cfdd39fad3547fe5ae0b58617924c8a"
server_key_version: "encryptionkey@0"
I20260926 06:15:13.378171 5520 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:15:13.379560 5535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:13.380043 5520 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:13.380250 5520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "3de72bcd61ce487cb81c681a1726de74"
format_stamp: "Formatted at 2026-09-26 06:15:13 on dist-test-slave-hndd"
server_key: "f6a3846dc6537f650fe4a1451fc021dd"
server_key_iv: "1cfdd39fad3547fe5ae0b58617924c8a"
server_key_version: "encryptionkey@0"
I20260926 06:15:13.380398 5520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:13.394956 5520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:13.397184 5520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:13.397397 5520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:13.402081 5520 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.254:35407
I20260926 06:15:13.402158 5587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.254:35407 every 8 connection(s)
I20260926 06:15:13.402601 5520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:15:13.404011 5588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:13.406910 5588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3de72bcd61ce487cb81c681a1726de74: Bootstrap starting.
I20260926 06:15:13.407524 5588 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3de72bcd61ce487cb81c681a1726de74: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:13.407787 5588 log.cc:824] T 00000000000000000000000000000000 P 3de72bcd61ce487cb81c681a1726de74: Log is configured to *not* fsync() on all Append() calls
I20260926 06:15:13.408468 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 5520
I20260926 06:15:13.408594 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/master-0/wal/instance
I20260926 06:15:13.408677 5588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3de72bcd61ce487cb81c681a1726de74: No bootstrap required, opened a new log
I20260926 06:15:13.408774 2491 external_mini_cluster.cc:1428] Setting key dc89ae47ec79554f25ce8b6f35ea0bf7
I20260926 06:15:13.411590 5588 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3de72bcd61ce487cb81c681a1726de74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de72bcd61ce487cb81c681a1726de74" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 35407 } }
I20260926 06:15:13.411836 5588 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3de72bcd61ce487cb81c681a1726de74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:13.411868 5588 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3de72bcd61ce487cb81c681a1726de74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3de72bcd61ce487cb81c681a1726de74, State: Initialized, Role: FOLLOWER
I20260926 06:15:13.412156 5588 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3de72bcd61ce487cb81c681a1726de74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de72bcd61ce487cb81c681a1726de74" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 35407 } }
I20260926 06:15:13.412294 5588 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3de72bcd61ce487cb81c681a1726de74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:15:13.412350 5588 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3de72bcd61ce487cb81c681a1726de74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:15:13.412412 5588 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3de72bcd61ce487cb81c681a1726de74 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:13.413156 5588 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3de72bcd61ce487cb81c681a1726de74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de72bcd61ce487cb81c681a1726de74" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 35407 } }
I20260926 06:15:13.413275 5588 leader_election.cc:302] T 00000000000000000000000000000000 P 3de72bcd61ce487cb81c681a1726de74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3de72bcd61ce487cb81c681a1726de74; no voters:
I20260926 06:15:13.413481 5588 leader_election.cc:288] T 00000000000000000000000000000000 P 3de72bcd61ce487cb81c681a1726de74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:15:13.413543 5593 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3de72bcd61ce487cb81c681a1726de74 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:15:13.413702 5593 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3de72bcd61ce487cb81c681a1726de74 [term 1 LEADER]: Becoming Leader. State: Replica: 3de72bcd61ce487cb81c681a1726de74, State: Running, Role: LEADER
I20260926 06:15:13.413762 5588 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3de72bcd61ce487cb81c681a1726de74 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:15:13.413792 5593 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3de72bcd61ce487cb81c681a1726de74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de72bcd61ce487cb81c681a1726de74" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 35407 } }
I20260926 06:15:13.414124 5593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3de72bcd61ce487cb81c681a1726de74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3de72bcd61ce487cb81c681a1726de74. Latest consensus state: current_term: 1 leader_uuid: "3de72bcd61ce487cb81c681a1726de74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de72bcd61ce487cb81c681a1726de74" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 35407 } } }
I20260926 06:15:13.414192 5593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3de72bcd61ce487cb81c681a1726de74 [sys.catalog]: This master's current role is: LEADER
I20260926 06:15:13.414377 5598 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:15:13.414839 5598 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:15:13.415352 5594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3de72bcd61ce487cb81c681a1726de74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3de72bcd61ce487cb81c681a1726de74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de72bcd61ce487cb81c681a1726de74" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 35407 } } }
I20260926 06:15:13.415795 5594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3de72bcd61ce487cb81c681a1726de74 [sys.catalog]: This master's current role is: LEADER
I20260926 06:15:13.416433 5598 catalog_manager.cc:1347] Generated new cluster ID: 5a03751a158147e39c7d71313c76dad5
I20260926 06:15:13.416484 5598 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 06:15:13 dist-test-slave-hndd krb5kdc[5507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:15:13.422274 5598 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:15:13.422408 5592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:13.409388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57474 (local address 127.2.110.254:35407)
0926 06:15:13.409585 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:13.409594 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:13.409633 (+ 39us) server_negotiation.cc:407] Connection header received
0926 06:15:13.410100 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:13.410107 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:13.410155 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:13.410269 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:13.411360 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:13.412886 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:13.414047 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:13.414691 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:13.414801 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:13.418244 (+ 3443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:13.418248 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:13.418248 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:13.418261 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:13.420353 (+ 2092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:13.420796 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:13.420800 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:13.420800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:13.420922 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:13.421152 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:13.421153 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:13.421154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:13.421519 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:13.421704 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:13.421981 (+ 277us) server_negotiation.cc:299] Negotiation successful
0926 06:15:13.422147 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":111,"threads_started":1}
I20260926 06:15:13.422916 5598 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:15:13.426491 5598 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3de72bcd61ce487cb81c681a1726de74: Generated new TSK 0
I20260926 06:15:13.426697 5598 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab.
Entry for principal kudu/127.2.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab.
I20260926 06:15:13.465637 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.193:0
--local_ip_for_outbound_sockets=127.2.110.193
--webserver_interface=127.2.110.193
--webserver_port=0
--tserver_master_addrs=127.2.110.254:35407
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab
--principal=kudu/127.2.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:33899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:15:13.566923 5614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:13.567168 5614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:13.567212 5614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:13.568439 5614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:13.568511 5614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:13.568569 5614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.193
I20260926 06:15:13.570214 5614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:33899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab
--principal=kudu/127.2.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.110.193
--webserver_port=0
--tserver_master_addrs=127.2.110.254:35407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.193
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:13.570489 5614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:13.570737 5614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:13.575937 5619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:13.576045 5622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:13.575945 5620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:13.576442 5614 server_base.cc:1034] running on GCE node
I20260926 06:15:13.576695 5614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:13.576960 5614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:13.578132 5614 hybrid_clock.cc:648] HybridClock initialized: now 1790403313578118 us; error 35 us; skew 500 ppm
Sep 26 06:15:13 dist-test-slave-hndd krb5kdc[5507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403313, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:13.580798 5614 init.cc:377] Logged in from keytab as kudu/127.2.110.193@KRBTEST.COM (short username kudu)
I20260926 06:15:13.581372 5614 webserver.cc:460] Webserver started at http://127.2.110.193:45253/ using document root <none> and password file <none>
I20260926 06:15:13.581687 5614 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:13.581763 5614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:13.581880 5614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:13.583184 5614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-0/data/instance:
uuid: "564befe9310b4c0798a1fa6fc43fd0f7"
format_stamp: "Formatted at 2026-09-26 06:15:13 on dist-test-slave-hndd"
server_key: "80a512f9ea8c0c6489d44ecbbb089d6d"
server_key_iv: "376ce5b4177c7087811b12c6144fb07a"
server_key_version: "encryptionkey@0"
I20260926 06:15:13.583658 5614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-0/wal/instance:
uuid: "564befe9310b4c0798a1fa6fc43fd0f7"
format_stamp: "Formatted at 2026-09-26 06:15:13 on dist-test-slave-hndd"
server_key: "80a512f9ea8c0c6489d44ecbbb089d6d"
server_key_iv: "376ce5b4177c7087811b12c6144fb07a"
server_key_version: "encryptionkey@0"
I20260926 06:15:13.585598 5614 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260926 06:15:13.586462 5629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:13.586633 5614 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:13.586736 5614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-0/wal
uuid: "564befe9310b4c0798a1fa6fc43fd0f7"
format_stamp: "Formatted at 2026-09-26 06:15:13 on dist-test-slave-hndd"
server_key: "80a512f9ea8c0c6489d44ecbbb089d6d"
server_key_iv: "376ce5b4177c7087811b12c6144fb07a"
server_key_version: "encryptionkey@0"
I20260926 06:15:13.586804 5614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:13.606958 5614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:13.609426 5614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:13.609556 5614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:13.609762 5614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:15:13.610100 5614 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:15:13.610137 5614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:13.610162 5614 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:15:13.610212 5614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:13.616501 5614 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.193:33915
I20260926 06:15:13.616570 5742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.193:33915 every 8 connection(s)
I20260926 06:15:13.616930 5614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
I20260926 06:15:13.622303 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 5614
I20260926 06:15:13.622396 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-0/wal/instance
I20260926 06:15:13.622540 2491 external_mini_cluster.cc:1428] Setting key aa8f38d3c0a6264ea3fe64e19122b747
Sep 26 06:15:13 dist-test-slave-hndd krb5kdc[5507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403313, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.193@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:15:13.628191 5592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:13.617764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:37599 (local address 127.2.110.254:35407)
0926 06:15:13.617843 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:13.617846 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:13.618079 (+ 233us) server_negotiation.cc:407] Connection header received
0926 06:15:13.618591 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:13.618592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:13.618606 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:13.618664 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:13.619636 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:13.620331 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:13.621534 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:13.621745 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:13.621813 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:13.624609 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:13.624611 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:13.624611 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:13.624621 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:13.626473 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:13.626982 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:13.626983 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:13.626984 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:13.627050 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:13.627383 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:13.627384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:13.627385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:13.627543 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:13.627628 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:13.627935 (+ 307us) server_negotiation.cc:299] Negotiation successful
0926 06:15:13.628045 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 06:15:13.628239 5745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:13.617726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35407 (local address 127.2.110.193:37599)
0926 06:15:13.617951 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:13.617991 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:13.618470 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:13.618718 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:13.618721 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:13.619315 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:13.619511 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:13.619516 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:13.620479 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:13.620480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:13.621418 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:13.621422 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:13.621833 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:13.621834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:13.621949 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:13.622583 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:13.622594 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:13.624500 (+ 1906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:13.626588 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:13.626595 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:13.626598 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:13.626890 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:13.627160 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:13.627161 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:13.627162 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:13.627306 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:13.627631 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:13.627637 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:13.627782 (+ 145us) client_negotiation.cc:769] Sending connection context
0926 06:15:13.627887 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 06:15:13.628045 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":121,"threads_started":1}
I20260926 06:15:13.628942 5743 heartbeater.cc:347] Connected to a master server at 127.2.110.254:35407
I20260926 06:15:13.629035 5743 heartbeater.cc:464] Registering TS with master...
I20260926 06:15:13.629257 5743 heartbeater.cc:511] Master 127.2.110.254:35407 requested a full tablet report, sending...
I20260926 06:15:13.629926 5552 ts_manager.cc:194] Registered new tserver with Master: 564befe9310b4c0798a1fa6fc43fd0f7 (127.2.110.193:33915)
I20260926 06:15:13.630910 5552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.193@KRBTEST.COM'} at 127.2.110.193:37599
WARNING: no policy specified for kudu/127.2.110.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab.
Entry for principal kudu/127.2.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab.
I20260926 06:15:13.648391 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.194:0
--local_ip_for_outbound_sockets=127.2.110.194
--webserver_interface=127.2.110.194
--webserver_port=0
--tserver_master_addrs=127.2.110.254:35407
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab
--principal=kudu/127.2.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:33899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:15:13.732049 5748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:13.732285 5748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:13.732311 5748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:13.733577 5748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:13.733628 5748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:13.733657 5748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.194
I20260926 06:15:13.735117 5748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:33899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab
--principal=kudu/127.2.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.110.194
--webserver_port=0
--tserver_master_addrs=127.2.110.254:35407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.194
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:13.735337 5748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:13.735522 5748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:13.740953 5754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:13.740937 5756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:13.740955 5753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:13.741241 5748 server_base.cc:1034] running on GCE node
I20260926 06:15:13.741505 5748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:13.741796 5748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:13.742985 5748 hybrid_clock.cc:648] HybridClock initialized: now 1790403313742963 us; error 57 us; skew 500 ppm
Sep 26 06:15:13 dist-test-slave-hndd krb5kdc[5507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403313, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:13.745735 5748 init.cc:377] Logged in from keytab as kudu/127.2.110.194@KRBTEST.COM (short username kudu)
I20260926 06:15:13.746490 5748 webserver.cc:460] Webserver started at http://127.2.110.194:37267/ using document root <none> and password file <none>
I20260926 06:15:13.746806 5748 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:13.746891 5748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:13.747077 5748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:13.748270 5748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-1/data/instance:
uuid: "540b137c84244f48a0d381e6217b92f5"
format_stamp: "Formatted at 2026-09-26 06:15:13 on dist-test-slave-hndd"
server_key: "8d4bf7cb60a7e46f8e0383208f3d39fb"
server_key_iv: "ceb70f00075b5a0970e1f8e4a3f156c0"
server_key_version: "encryptionkey@0"
I20260926 06:15:13.748673 5748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-1/wal/instance:
uuid: "540b137c84244f48a0d381e6217b92f5"
format_stamp: "Formatted at 2026-09-26 06:15:13 on dist-test-slave-hndd"
server_key: "8d4bf7cb60a7e46f8e0383208f3d39fb"
server_key_iv: "ceb70f00075b5a0970e1f8e4a3f156c0"
server_key_version: "encryptionkey@0"
I20260926 06:15:13.750291 5748 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260926 06:15:13.751185 5763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:13.751384 5748 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:13.751507 5748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-1/wal
uuid: "540b137c84244f48a0d381e6217b92f5"
format_stamp: "Formatted at 2026-09-26 06:15:13 on dist-test-slave-hndd"
server_key: "8d4bf7cb60a7e46f8e0383208f3d39fb"
server_key_iv: "ceb70f00075b5a0970e1f8e4a3f156c0"
server_key_version: "encryptionkey@0"
I20260926 06:15:13.751591 5748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:13.767020 5748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:13.769109 5748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:13.769286 5748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:13.769568 5748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:15:13.769891 5748 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:15:13.769948 5748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:13.769985 5748 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:15:13.770035 5748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:13.776733 5748 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.194:46029
I20260926 06:15:13.776826 5876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.194:46029 every 8 connection(s)
I20260926 06:15:13.777184 5748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
I20260926 06:15:13.783910 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 5748
I20260926 06:15:13.784034 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-1/wal/instance
I20260926 06:15:13.784227 2491 external_mini_cluster.cc:1428] Setting key a761dde14a8dce45a429a90aa51713d1
Sep 26 06:15:13 dist-test-slave-hndd krb5kdc[5507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403313, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.194@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:15:13.788712 5592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:13.777981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:50079 (local address 127.2.110.254:35407)
0926 06:15:13.778054 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:13.778056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:13.778306 (+ 250us) server_negotiation.cc:407] Connection header received
0926 06:15:13.778773 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:13.778774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:13.778786 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:13.778867 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:13.779811 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:13.780383 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:13.781943 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:13.782234 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:13.782298 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:13.785214 (+ 2916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:13.785215 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:13.785216 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:13.785223 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:13.787141 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:13.787574 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:13.787576 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:13.787576 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:13.787658 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:13.787982 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:13.787983 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:13.787984 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:13.788125 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:13.788206 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:13.788517 (+ 311us) server_negotiation.cc:299] Negotiation successful
0926 06:15:13.788617 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 06:15:13.788971 5879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:13.777954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35407 (local address 127.2.110.194:50079)
0926 06:15:13.778147 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:13.778192 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:13.778678 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:13.778933 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:13.778936 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:13.779506 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:13.779711 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:13.779716 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:13.780580 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:13.780582 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:13.781794 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:13.781800 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:13.782347 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:13.782348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:13.782507 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:13.783121 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:13.783130 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:13.785060 (+ 1930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:13.787246 (+ 2186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:13.787252 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:13.787254 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:13.787489 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:13.787765 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:13.787766 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:13.787766 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:13.787879 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:13.788208 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:13.788215 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:13.788377 (+ 162us) client_negotiation.cc:769] Sending connection context
0926 06:15:13.788498 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 06:15:13.788677 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":120,"threads_started":1}
I20260926 06:15:13.789456 5877 heartbeater.cc:347] Connected to a master server at 127.2.110.254:35407
I20260926 06:15:13.789554 5877 heartbeater.cc:464] Registering TS with master...
I20260926 06:15:13.789763 5877 heartbeater.cc:511] Master 127.2.110.254:35407 requested a full tablet report, sending...
I20260926 06:15:13.790154 5552 ts_manager.cc:194] Registered new tserver with Master: 540b137c84244f48a0d381e6217b92f5 (127.2.110.194:46029)
I20260926 06:15:13.790561 5552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.194@KRBTEST.COM'} at 127.2.110.194:50079
WARNING: no policy specified for kudu/127.2.110.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab.
Entry for principal kudu/127.2.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab.
I20260926 06:15:13.811930 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.195:0
--local_ip_for_outbound_sockets=127.2.110.195
--webserver_interface=127.2.110.195
--webserver_port=0
--tserver_master_addrs=127.2.110.254:35407
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab
--principal=kudu/127.2.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:33899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:15:13.899029 5882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:13.899276 5882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:13.899303 5882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:13.900517 5882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:13.900560 5882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:13.900702 5882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.195
I20260926 06:15:13.903455 5882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:33899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab
--principal=kudu/127.2.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.110.195
--webserver_port=0
--tserver_master_addrs=127.2.110.254:35407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.195
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:13.903754 5882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:13.903997 5882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:13.910231 5888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:13.910269 5887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:13.910301 5890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:13.910529 5882 server_base.cc:1034] running on GCE node
I20260926 06:15:13.910737 5882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:13.911002 5882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:13.912190 5882 hybrid_clock.cc:648] HybridClock initialized: now 1790403313912172 us; error 33 us; skew 500 ppm
Sep 26 06:15:13 dist-test-slave-hndd krb5kdc[5507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403313, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:13.915136 5882 init.cc:377] Logged in from keytab as kudu/127.2.110.195@KRBTEST.COM (short username kudu)
I20260926 06:15:13.915649 5882 webserver.cc:460] Webserver started at http://127.2.110.195:42527/ using document root <none> and password file <none>
I20260926 06:15:13.915903 5882 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:13.915975 5882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:13.916124 5882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:13.918219 5882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-2/data/instance:
uuid: "b6f404d763354e4ea32771218a69ce7d"
format_stamp: "Formatted at 2026-09-26 06:15:13 on dist-test-slave-hndd"
server_key: "1e69c36af67b31b65693baf4444654d2"
server_key_iv: "983ce5fb185f0c9fbbc4980af5092efc"
server_key_version: "encryptionkey@0"
I20260926 06:15:13.918754 5882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-2/wal/instance:
uuid: "b6f404d763354e4ea32771218a69ce7d"
format_stamp: "Formatted at 2026-09-26 06:15:13 on dist-test-slave-hndd"
server_key: "1e69c36af67b31b65693baf4444654d2"
server_key_iv: "983ce5fb185f0c9fbbc4980af5092efc"
server_key_version: "encryptionkey@0"
I20260926 06:15:13.920321 5882 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:15:13.921315 5897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:13.921574 5882 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:13.921689 5882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-2/wal
uuid: "b6f404d763354e4ea32771218a69ce7d"
format_stamp: "Formatted at 2026-09-26 06:15:13 on dist-test-slave-hndd"
server_key: "1e69c36af67b31b65693baf4444654d2"
server_key_iv: "983ce5fb185f0c9fbbc4980af5092efc"
server_key_version: "encryptionkey@0"
I20260926 06:15:13.921782 5882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:13.939069 5882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:13.941618 5882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:13.941795 5882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:13.942054 5882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:15:13.942379 5882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:15:13.942435 5882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:13.942487 5882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:15:13.942535 5882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:13.948796 5882 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.195:33269
I20260926 06:15:13.949055 6010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.195:33269 every 8 connection(s)
I20260926 06:15:13.949218 5882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
Sep 26 06:15:13 dist-test-slave-hndd krb5kdc[5507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403313, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.195@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:15:13.958148 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 5882
I20260926 06:15:13.958250 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403292369990-2491-0/minicluster-data/ts-2/wal/instance
I20260926 06:15:13.958447 2491 external_mini_cluster.cc:1428] Setting key 3443e940dc511b9c7cb990de6e6c7ef8
I20260926 06:15:13.960019 5592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:13.950087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:44325 (local address 127.2.110.254:35407)
0926 06:15:13.950281 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:13.950284 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:13.950450 (+ 166us) server_negotiation.cc:407] Connection header received
0926 06:15:13.951042 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:13.951043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:13.951057 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:13.951112 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:13.952108 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:13.952632 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:13.954149 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:13.954366 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:13.954457 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:13.956795 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:13.956796 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:13.956797 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:13.956804 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:13.958327 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:13.958903 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:13.958903 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:13.958904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:13.958984 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:13.959314 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:13.959315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:13.959315 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:13.959468 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:13.959567 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:13.959885 (+ 318us) server_negotiation.cc:299] Negotiation successful
0926 06:15:13.959954 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 06:15:13.960300 6013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:13.949958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35407 (local address 127.2.110.195:44325)
0926 06:15:13.950285 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:13.950338 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:13.950907 (+ 569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:13.951188 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:13.951193 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:13.951797 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:13.952004 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:13.952008 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:13.952736 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:13.952737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:13.953999 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:13.954002 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:13.954466 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:13.954467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:13.954644 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:13.955158 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:13.955166 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:13.956697 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:13.958439 (+ 1742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:13.958446 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:13.958448 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:13.958810 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:13.959055 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:13.959055 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:13.959056 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:13.959236 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:13.959568 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:13.959574 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:13.959754 (+ 180us) client_negotiation.cc:769] Sending connection context
0926 06:15:13.959861 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 06:15:13.960021 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":108,"threads_started":1}
I20260926 06:15:13.960780 6011 heartbeater.cc:347] Connected to a master server at 127.2.110.254:35407
I20260926 06:15:13.960891 6011 heartbeater.cc:464] Registering TS with master...
I20260926 06:15:13.961095 6011 heartbeater.cc:511] Master 127.2.110.254:35407 requested a full tablet report, sending...
I20260926 06:15:13.961480 5552 ts_manager.cc:194] Registered new tserver with Master: b6f404d763354e4ea32771218a69ce7d (127.2.110.195:33269)
I20260926 06:15:13.961859 5552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.195@KRBTEST.COM'} at 127.2.110.195:44325
I20260926 06:15:13.970482 2491 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 06:15:13 dist-test-slave-hndd krb5kdc[5507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403313, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 06:15:13 dist-test-slave-hndd krb5kdc[5507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403313, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:15:13.996961 5592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:13.987858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57482 (local address 127.2.110.254:35407)
0926 06:15:13.987959 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:13.987961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:13.988056 (+ 95us) server_negotiation.cc:407] Connection header received
0926 06:15:13.988145 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:13.988146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:13.988157 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:13.988215 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:13.989026 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:13.989641 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:13.990651 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:13.990837 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:13.990904 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:13.993943 (+ 3039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:13.993946 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:13.993946 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:13.993957 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:13.995422 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:13.995800 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:13.995800 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:13.995801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:13.995857 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:13.996186 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:13.996193 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:13.996194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:13.996444 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:13.996560 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:13.996773 (+ 213us) server_negotiation.cc:299] Negotiation successful
0926 06:15:13.996871 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260926 06:15:13.997380 5552 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:57482
I20260926 06:15:13.997989 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 5614
I20260926 06:15:14.003105 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 5748
I20260926 06:15:14.008730 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 5882
I20260926 06:15:14.013916 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 5520
2026-09-26T06:15:14Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1929 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:15:14 dist-test-slave-hndd krb5kdc[6023](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:15:14 dist-test-slave-hndd krb5kdc[6023](info): set up 2 sockets
Sep 26 06:15:14 dist-test-slave-hndd krb5kdc[6023](info): commencing operation
krb5kdc: starting...
W20260926 06:15:15.077927 2491 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Sep 26 06:15:15 dist-test-slave-hndd krb5kdc[6023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:15:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:15:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
I20260926 06:15:15.208614 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:34755
--webserver_interface=127.2.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:37395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.110.254:34755
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:15:15.292387 6036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:15.292677 6036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:15.292721 6036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:15.294096 6036 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:15:15.294174 6036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:15.294198 6036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:15.294229 6036 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:15:15.294260 6036 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:15:15.295939 6036 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:37395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.110.254:34755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:34755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:15.296260 6036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:15.296511 6036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:15.302017 6042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:15.302122 6044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:15.302057 6036 server_base.cc:1034] running on GCE node
W20260926 06:15:15.302249 6041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:15.302651 6036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:15.303192 6036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:15.304400 6036 hybrid_clock.cc:648] HybridClock initialized: now 1790403315304384 us; error 55 us; skew 500 ppm
Sep 26 06:15:15 dist-test-slave-hndd krb5kdc[6023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403315, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:15.310735 6036 init.cc:377] Logged in from keytab as kudu/127.2.110.254@KRBTEST.COM (short username kudu)
I20260926 06:15:15.311188 6036 webserver.cc:460] Webserver started at http://127.2.110.254:44593/ using document root <none> and password file <none>
I20260926 06:15:15.311398 6036 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:15.311439 6036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:15.311537 6036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:15.312597 6036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/master-0/data/instance:
uuid: "8cbcf06146d7467db24ef3180c4efbcd"
format_stamp: "Formatted at 2026-09-26 06:15:15 on dist-test-slave-hndd"
server_key: "412fb138e29b83950b8f6df8150de730"
server_key_iv: "6a81a189b0710facbfa465171d82cccb"
server_key_version: "encryptionkey@0"
I20260926 06:15:15.312915 6036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/master-0/wal/instance:
uuid: "8cbcf06146d7467db24ef3180c4efbcd"
format_stamp: "Formatted at 2026-09-26 06:15:15 on dist-test-slave-hndd"
server_key: "412fb138e29b83950b8f6df8150de730"
server_key_iv: "6a81a189b0710facbfa465171d82cccb"
server_key_version: "encryptionkey@0"
I20260926 06:15:15.314373 6036 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:15:15.315100 6051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:15.315280 6036 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:15.315374 6036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "8cbcf06146d7467db24ef3180c4efbcd"
format_stamp: "Formatted at 2026-09-26 06:15:15 on dist-test-slave-hndd"
server_key: "412fb138e29b83950b8f6df8150de730"
server_key_iv: "6a81a189b0710facbfa465171d82cccb"
server_key_version: "encryptionkey@0"
I20260926 06:15:15.315435 6036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:15.341536 6036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:15.343979 6036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:15.344134 6036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:15.348300 6036 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.254:34755
I20260926 06:15:15.348338 6103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.254:34755 every 8 connection(s)
I20260926 06:15:15.348897 6036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:15:15.349735 6104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:15.352219 6104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cbcf06146d7467db24ef3180c4efbcd: Bootstrap starting.
I20260926 06:15:15.352862 6104 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8cbcf06146d7467db24ef3180c4efbcd: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:15.353176 6104 log.cc:824] T 00000000000000000000000000000000 P 8cbcf06146d7467db24ef3180c4efbcd: Log is configured to *not* fsync() on all Append() calls
I20260926 06:15:15.354405 6104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cbcf06146d7467db24ef3180c4efbcd: No bootstrap required, opened a new log
I20260926 06:15:15.354410 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 6036
I20260926 06:15:15.354521 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/master-0/wal/instance
I20260926 06:15:15.354732 2491 external_mini_cluster.cc:1428] Setting key 6b059b12c8b1a9bf21a547d23f27cd1a
I20260926 06:15:15.357517 6104 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8cbcf06146d7467db24ef3180c4efbcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cbcf06146d7467db24ef3180c4efbcd" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 34755 } }
I20260926 06:15:15.357744 6104 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8cbcf06146d7467db24ef3180c4efbcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:15.357790 6104 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8cbcf06146d7467db24ef3180c4efbcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cbcf06146d7467db24ef3180c4efbcd, State: Initialized, Role: FOLLOWER
I20260926 06:15:15.357939 6104 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8cbcf06146d7467db24ef3180c4efbcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cbcf06146d7467db24ef3180c4efbcd" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 34755 } }
I20260926 06:15:15.358026 6104 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8cbcf06146d7467db24ef3180c4efbcd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:15:15.358063 6104 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8cbcf06146d7467db24ef3180c4efbcd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:15:15.358116 6104 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8cbcf06146d7467db24ef3180c4efbcd [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:15.358959 6104 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8cbcf06146d7467db24ef3180c4efbcd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cbcf06146d7467db24ef3180c4efbcd" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 34755 } }
I20260926 06:15:15.359091 6104 leader_election.cc:302] T 00000000000000000000000000000000 P 8cbcf06146d7467db24ef3180c4efbcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8cbcf06146d7467db24ef3180c4efbcd; no voters:
I20260926 06:15:15.359344 6104 leader_election.cc:288] T 00000000000000000000000000000000 P 8cbcf06146d7467db24ef3180c4efbcd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:15:15.359376 6109 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8cbcf06146d7467db24ef3180c4efbcd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:15:15.359515 6109 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8cbcf06146d7467db24ef3180c4efbcd [term 1 LEADER]: Becoming Leader. State: Replica: 8cbcf06146d7467db24ef3180c4efbcd, State: Running, Role: LEADER
I20260926 06:15:15.359608 6109 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8cbcf06146d7467db24ef3180c4efbcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cbcf06146d7467db24ef3180c4efbcd" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 34755 } }
I20260926 06:15:15.359685 6104 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8cbcf06146d7467db24ef3180c4efbcd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:15:15.360062 6109 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cbcf06146d7467db24ef3180c4efbcd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8cbcf06146d7467db24ef3180c4efbcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cbcf06146d7467db24ef3180c4efbcd" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 34755 } } }
I20260926 06:15:15.360157 6109 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cbcf06146d7467db24ef3180c4efbcd [sys.catalog]: This master's current role is: LEADER
I20260926 06:15:15.360306 6109 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cbcf06146d7467db24ef3180c4efbcd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8cbcf06146d7467db24ef3180c4efbcd. Latest consensus state: current_term: 1 leader_uuid: "8cbcf06146d7467db24ef3180c4efbcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cbcf06146d7467db24ef3180c4efbcd" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 34755 } } }
I20260926 06:15:15.360415 6114 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:15:15.360482 6109 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cbcf06146d7467db24ef3180c4efbcd [sys.catalog]: This master's current role is: LEADER
I20260926 06:15:15.361027 6114 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:15:15.363046 6114 catalog_manager.cc:1347] Generated new cluster ID: 267ef132b0ff4dd3b0109498b44b8679
I20260926 06:15:15.363169 6114 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 06:15:15 dist-test-slave-hndd krb5kdc[6023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:15:15.369503 6108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:15.355454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46104 (local address 127.2.110.254:34755)
0926 06:15:15.355693 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:15.355703 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:15.355743 (+ 40us) server_negotiation.cc:407] Connection header received
0926 06:15:15.356466 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:15.356476 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:15.356544 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:15.356728 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:15.357642 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:15.358979 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:15.360303 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:15.360952 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:15.361052 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:15.364188 (+ 3136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:15.364192 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:15.364192 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:15.364206 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:15.366758 (+ 2552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:15.367332 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:15.367336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:15.367337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:15.367473 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:15.367713 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:15.367714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:15.367714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:15.368201 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:15.368403 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:15.369032 (+ 629us) server_negotiation.cc:299] Negotiation successful
0926 06:15:15.369207 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":119,"threads_started":1}
I20260926 06:15:15.387498 6114 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:15:15.388404 6114 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:15:15.399406 6114 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8cbcf06146d7467db24ef3180c4efbcd: Generated new TSK 0
I20260926 06:15:15.399688 6114 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab.
Entry for principal kudu/127.2.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab.
I20260926 06:15:15.436317 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.193:0
--local_ip_for_outbound_sockets=127.2.110.193
--webserver_interface=127.2.110.193
--webserver_port=0
--tserver_master_addrs=127.2.110.254:34755
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab
--principal=kudu/127.2.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:37395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:15:15.543023 6130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:15.543242 6130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:15.543269 6130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:15.544438 6130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:15.544478 6130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:15.544505 6130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.193
I20260926 06:15:15.546124 6130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:37395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab
--principal=kudu/127.2.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.110.193
--webserver_port=0
--tserver_master_addrs=127.2.110.254:34755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.193
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:15.546341 6130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:15.546535 6130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:15.553531 6135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:15.553602 6136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:15.553660 6138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:15.554025 6130 server_base.cc:1034] running on GCE node
I20260926 06:15:15.554289 6130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:15.554602 6130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:15.555768 6130 hybrid_clock.cc:648] HybridClock initialized: now 1790403315555746 us; error 39 us; skew 500 ppm
Sep 26 06:15:15 dist-test-slave-hndd krb5kdc[6023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403315, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:15.558902 6130 init.cc:377] Logged in from keytab as kudu/127.2.110.193@KRBTEST.COM (short username kudu)
I20260926 06:15:15.559502 6130 webserver.cc:460] Webserver started at http://127.2.110.193:38743/ using document root <none> and password file <none>
I20260926 06:15:15.559778 6130 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:15.559885 6130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:15.559998 6130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:15.561161 6130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/ts-0/data/instance:
uuid: "fcbff60f21354654a8e0190ce51398db"
format_stamp: "Formatted at 2026-09-26 06:15:15 on dist-test-slave-hndd"
server_key: "c1fcc9bfaf202ba8c3305d949e8ef197"
server_key_iv: "6a40577940e30913a8dbccbdde7cdf25"
server_key_version: "encryptionkey@0"
I20260926 06:15:15.561602 6130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/ts-0/wal/instance:
uuid: "fcbff60f21354654a8e0190ce51398db"
format_stamp: "Formatted at 2026-09-26 06:15:15 on dist-test-slave-hndd"
server_key: "c1fcc9bfaf202ba8c3305d949e8ef197"
server_key_iv: "6a40577940e30913a8dbccbdde7cdf25"
server_key_version: "encryptionkey@0"
I20260926 06:15:15.563099 6130 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:15:15.564009 6145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:15.564258 6130 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:15:15.564365 6130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/ts-0/wal
uuid: "fcbff60f21354654a8e0190ce51398db"
format_stamp: "Formatted at 2026-09-26 06:15:15 on dist-test-slave-hndd"
server_key: "c1fcc9bfaf202ba8c3305d949e8ef197"
server_key_iv: "6a40577940e30913a8dbccbdde7cdf25"
server_key_version: "encryptionkey@0"
I20260926 06:15:15.564455 6130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:15.574955 6130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:15.577191 6130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:15.577412 6130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:15.577682 6130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:15:15.578028 6130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:15:15.578090 6130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:15.578138 6130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:15:15.578171 6130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:15.584537 6130 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.193:35951
I20260926 06:15:15.584580 6258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.193:35951 every 8 connection(s)
I20260926 06:15:15.584940 6130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
Sep 26 06:15:15 dist-test-slave-hndd krb5kdc[6023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403315, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.193@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:15:15.593763 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 6130
I20260926 06:15:15.593883 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/ts-0/wal/instance
I20260926 06:15:15.594091 2491 external_mini_cluster.cc:1428] Setting key ebd6e395850a0182e91a77beb4a4dbbd
I20260926 06:15:15.596469 6108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:15.585782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:47247 (local address 127.2.110.254:34755)
0926 06:15:15.585872 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:15.585874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:15.586248 (+ 374us) server_negotiation.cc:407] Connection header received
0926 06:15:15.587405 (+ 1157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:15.587406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:15.587422 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:15.587489 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:15.588191 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:15.588708 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:15.589791 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:15.589998 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:15.590090 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:15.592625 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:15.592627 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:15.592627 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:15.592636 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:15.594636 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:15.595149 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:15.595151 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:15.595151 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:15.595220 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:15.595566 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:15.595567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:15.595568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:15.595790 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:15.595899 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:15.596203 (+ 304us) server_negotiation.cc:299] Negotiation successful
0926 06:15:15.596310 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 06:15:15.596485 6261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:15.585749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34755 (local address 127.2.110.193:47247)
0926 06:15:15.586079 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:15.586122 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:15.587279 (+ 1157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:15.587567 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:15.587570 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:15.587861 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:15.588071 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:15.588075 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:15.588813 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:15.588815 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:15.589691 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:15.589693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:15.590103 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:15.590104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:15.590247 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:15.590841 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:15.590851 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:15.592495 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:15.594743 (+ 2248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:15.594751 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:15.594753 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:15.595060 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:15.595298 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:15.595299 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:15.595299 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:15.595474 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:15.595894 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:15.595902 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:15.596063 (+ 161us) client_negotiation.cc:769] Sending connection context
0926 06:15:15.596160 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 06:15:15.596299 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":225,"threads_started":1}
I20260926 06:15:15.597646 6259 heartbeater.cc:347] Connected to a master server at 127.2.110.254:34755
I20260926 06:15:15.597818 6259 heartbeater.cc:464] Registering TS with master...
I20260926 06:15:15.598107 6259 heartbeater.cc:511] Master 127.2.110.254:34755 requested a full tablet report, sending...
I20260926 06:15:15.598838 6068 ts_manager.cc:194] Registered new tserver with Master: fcbff60f21354654a8e0190ce51398db (127.2.110.193:35951)
I20260926 06:15:15.599521 6068 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.193@KRBTEST.COM'} at 127.2.110.193:47247
WARNING: no policy specified for kudu/127.2.110.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab.
Entry for principal kudu/127.2.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab.
I20260926 06:15:15.620517 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.194:0
--local_ip_for_outbound_sockets=127.2.110.194
--webserver_interface=127.2.110.194
--webserver_port=0
--tserver_master_addrs=127.2.110.254:34755
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab
--principal=kudu/127.2.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:37395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:15:15.705328 6264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:15.705618 6264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:15.705646 6264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:15.706866 6264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:15.706928 6264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:15.706959 6264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.194
I20260926 06:15:15.708536 6264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:37395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab
--principal=kudu/127.2.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.110.194
--webserver_port=0
--tserver_master_addrs=127.2.110.254:34755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.194
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:15.708808 6264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:15.709074 6264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:15.714375 6270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:15.714380 6269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:15.714494 6272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:15.714655 6264 server_base.cc:1034] running on GCE node
I20260926 06:15:15.714926 6264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:15.715167 6264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:15.716348 6264 hybrid_clock.cc:648] HybridClock initialized: now 1790403315716339 us; error 32 us; skew 500 ppm
Sep 26 06:15:15 dist-test-slave-hndd krb5kdc[6023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403315, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:15.719103 6264 init.cc:377] Logged in from keytab as kudu/127.2.110.194@KRBTEST.COM (short username kudu)
I20260926 06:15:15.719595 6264 webserver.cc:460] Webserver started at http://127.2.110.194:41973/ using document root <none> and password file <none>
I20260926 06:15:15.719791 6264 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:15.719831 6264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:15.719921 6264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:15.720916 6264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/ts-1/data/instance:
uuid: "e744a81ef87e43dcbdc147ac8eb3dcbb"
format_stamp: "Formatted at 2026-09-26 06:15:15 on dist-test-slave-hndd"
server_key: "d1e897c197fc5be14d5bf1b241fdcebe"
server_key_iv: "317ec2b58a175eeebba7ad76cdf0106a"
server_key_version: "encryptionkey@0"
I20260926 06:15:15.721238 6264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/ts-1/wal/instance:
uuid: "e744a81ef87e43dcbdc147ac8eb3dcbb"
format_stamp: "Formatted at 2026-09-26 06:15:15 on dist-test-slave-hndd"
server_key: "d1e897c197fc5be14d5bf1b241fdcebe"
server_key_iv: "317ec2b58a175eeebba7ad76cdf0106a"
server_key_version: "encryptionkey@0"
I20260926 06:15:15.722615 6264 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:15:15.723344 6279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:15.723528 6264 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:15:15.723631 6264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/ts-1/wal
uuid: "e744a81ef87e43dcbdc147ac8eb3dcbb"
format_stamp: "Formatted at 2026-09-26 06:15:15 on dist-test-slave-hndd"
server_key: "d1e897c197fc5be14d5bf1b241fdcebe"
server_key_iv: "317ec2b58a175eeebba7ad76cdf0106a"
server_key_version: "encryptionkey@0"
I20260926 06:15:15.723690 6264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:15.734091 6264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:15.736685 6264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:15.736815 6264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:15.737048 6264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:15:15.737469 6264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:15:15.737516 6264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:15.737541 6264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:15:15.737586 6264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:15.743527 6264 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.194:39385
I20260926 06:15:15.743736 6392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.194:39385 every 8 connection(s)
I20260926 06:15:15.743973 6264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
I20260926 06:15:15.745790 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 6264
I20260926 06:15:15.745925 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/ts-1/wal/instance
I20260926 06:15:15.746112 2491 external_mini_cluster.cc:1428] Setting key fbc2bdebbdd671cb6771db986bd7e494
Sep 26 06:15:15 dist-test-slave-hndd krb5kdc[6023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403315, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.194@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.110.195@KRBTEST.COM; defaulting to no policy
I20260926 06:15:15.756513 6108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:15.744779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:37675 (local address 127.2.110.254:34755)
0926 06:15:15.744834 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:15.744836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:15.745034 (+ 198us) server_negotiation.cc:407] Connection header received
0926 06:15:15.746197 (+ 1163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:15.746199 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:15.746213 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:15.746301 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:15.747202 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:15.747967 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:15.749196 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:15.749549 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:15.749620 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:15.752610 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:15.752612 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:15.752613 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:15.752623 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:15.754637 (+ 2014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:15.755123 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:15.755124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:15.755124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:15.755191 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:15.755547 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:15.755549 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:15.755549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:15.755710 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:15.755772 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:15.756273 (+ 501us) server_negotiation.cc:299] Negotiation successful
0926 06:15:15.756419 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 06:15:15.756538 6395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:15.744734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34755 (local address 127.2.110.194:37675)
0926 06:15:15.744900 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:15.744944 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:15.746068 (+ 1124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:15.746399 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:15.746403 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:15.746703 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:15.746986 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:15.746991 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:15.748101 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:15.748102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:15.749095 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:15.749098 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:15.749657 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:15.749658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:15.749770 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:15.750424 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:15.750438 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:15.752498 (+ 2060us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:15.754745 (+ 2247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:15.754754 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:15.754756 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:15.755036 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:15.755288 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:15.755289 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:15.755289 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:15.755471 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:15.755800 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:15.755807 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:15.755968 (+ 161us) client_negotiation.cc:769] Sending connection context
0926 06:15:15.756076 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 06:15:15.756290 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":93,"threads_started":1}
I20260926 06:15:15.757223 6393 heartbeater.cc:347] Connected to a master server at 127.2.110.254:34755
I20260926 06:15:15.757316 6393 heartbeater.cc:464] Registering TS with master...
I20260926 06:15:15.757545 6393 heartbeater.cc:511] Master 127.2.110.254:34755 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.195@KRBTEST.COM" created.
I20260926 06:15:15.758080 6068 ts_manager.cc:194] Registered new tserver with Master: e744a81ef87e43dcbdc147ac8eb3dcbb (127.2.110.194:39385)
I20260926 06:15:15.758587 6068 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.194@KRBTEST.COM'} at 127.2.110.194:37675
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab.
Entry for principal kudu/127.2.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab.
I20260926 06:15:15.774456 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.195:0
--local_ip_for_outbound_sockets=127.2.110.195
--webserver_interface=127.2.110.195
--webserver_port=0
--tserver_master_addrs=127.2.110.254:34755
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab
--principal=kudu/127.2.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:37395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:15:15.858991 6398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:15.859256 6398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:15.859283 6398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:15.860633 6398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:15.860687 6398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:15.860718 6398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.195
I20260926 06:15:15.862511 6398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:37395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab
--principal=kudu/127.2.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.110.195
--webserver_port=0
--tserver_master_addrs=127.2.110.254:34755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.195
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:15.862777 6398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:15.863020 6398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:15.868217 6406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:15.868212 6403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:15.868217 6404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:15.868348 6398 server_base.cc:1034] running on GCE node
I20260926 06:15:15.868619 6398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:15.868933 6398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:15.870115 6398 hybrid_clock.cc:648] HybridClock initialized: now 1790403315870103 us; error 60 us; skew 500 ppm
Sep 26 06:15:15 dist-test-slave-hndd krb5kdc[6023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403315, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:15.872619 6398 init.cc:377] Logged in from keytab as kudu/127.2.110.195@KRBTEST.COM (short username kudu)
I20260926 06:15:15.873095 6398 webserver.cc:460] Webserver started at http://127.2.110.195:33833/ using document root <none> and password file <none>
I20260926 06:15:15.873371 6398 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:15.873458 6398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:15.873606 6398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:15.874750 6398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/ts-2/data/instance:
uuid: "0adc87bbeb2c489b8222c330a0fc885a"
format_stamp: "Formatted at 2026-09-26 06:15:15 on dist-test-slave-hndd"
server_key: "2263afa3c236e7bbddd44e364de05c79"
server_key_iv: "073995ee204b2bd5f3af416118f4df51"
server_key_version: "encryptionkey@0"
I20260926 06:15:15.875316 6398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/ts-2/wal/instance:
uuid: "0adc87bbeb2c489b8222c330a0fc885a"
format_stamp: "Formatted at 2026-09-26 06:15:15 on dist-test-slave-hndd"
server_key: "2263afa3c236e7bbddd44e364de05c79"
server_key_iv: "073995ee204b2bd5f3af416118f4df51"
server_key_version: "encryptionkey@0"
I20260926 06:15:15.876891 6398 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:15:15.877786 6413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:15.877995 6398 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 06:15:15.878109 6398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/ts-2/wal
uuid: "0adc87bbeb2c489b8222c330a0fc885a"
format_stamp: "Formatted at 2026-09-26 06:15:15 on dist-test-slave-hndd"
server_key: "2263afa3c236e7bbddd44e364de05c79"
server_key_iv: "073995ee204b2bd5f3af416118f4df51"
server_key_version: "encryptionkey@0"
I20260926 06:15:15.878199 6398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:15.890306 6398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:15.893196 6398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:15.893401 6398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:15.893682 6398 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:15:15.894116 6398 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:15:15.894166 6398 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:15.894217 6398 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:15:15.894246 6398 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:15.902123 6398 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.195:39089
I20260926 06:15:15.902208 6526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.195:39089 every 8 connection(s)
I20260926 06:15:15.902449 6398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
Sep 26 06:15:15 dist-test-slave-hndd krb5kdc[6023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403315, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.195@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:15:15.909921 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 6398
I20260926 06:15:15.910054 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403292369990-2491-0/minicluster-data/ts-2/wal/instance
I20260926 06:15:15.910287 2491 external_mini_cluster.cc:1428] Setting key 08498589e81ccd91f7fe641c67ca7653
I20260926 06:15:15.912904 6108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:15.903193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:41615 (local address 127.2.110.254:34755)
0926 06:15:15.903272 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:15.903275 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:15.903506 (+ 231us) server_negotiation.cc:407] Connection header received
0926 06:15:15.904470 (+ 964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:15.904471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:15.904485 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:15.904540 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:15.905194 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:15.905789 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:15.906827 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:15.907039 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:15.907127 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:15.909301 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:15.909303 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:15.909303 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:15.909313 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:15.911359 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:15.911829 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:15.911830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:15.911831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:15.911895 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:15.912181 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:15.912182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:15.912182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:15.912316 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:15.912408 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:15.912697 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 06:15:15.912790 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 06:15:15.913190 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:15.903206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34755 (local address 127.2.110.195:41615)
0926 06:15:15.903379 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:15.903415 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:15.904347 (+ 932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:15.904628 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:15.904632 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:15.904899 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:15.905099 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:15.905103 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:15.905901 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:15.905902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:15.906719 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:15.906721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:15.907138 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:15.907140 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:15.907268 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:15.907814 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:15.907822 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:15.909200 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:15.911465 (+ 2265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:15.911472 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:15.911475 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:15.911729 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:15.911983 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:15.911984 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:15.911984 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:15.912104 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:15.912390 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:15.912396 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:15.912554 (+ 158us) client_negotiation.cc:769] Sending connection context
0926 06:15:15.912676 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 06:15:15.912889 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":83,"threads_started":1}
I20260926 06:15:15.913761 6527 heartbeater.cc:347] Connected to a master server at 127.2.110.254:34755
I20260926 06:15:15.913851 6527 heartbeater.cc:464] Registering TS with master...
I20260926 06:15:15.914005 6527 heartbeater.cc:511] Master 127.2.110.254:34755 requested a full tablet report, sending...
I20260926 06:15:15.914372 6068 ts_manager.cc:194] Registered new tserver with Master: 0adc87bbeb2c489b8222c330a0fc885a (127.2.110.195:39089)
I20260926 06:15:15.914740 6068 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.195@KRBTEST.COM'} at 127.2.110.195:41615
I20260926 06:15:15.922238 2491 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 06:15:15 dist-test-slave-hndd krb5kdc[6023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 06:15:15 dist-test-slave-hndd krb5kdc[6023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.110.193@KRBTEST.COM
I20260926 06:15:15.946271 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:15.938119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59390 (local address 127.2.110.193:35951)
0926 06:15:15.938299 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:15.938304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:15.938316 (+ 12us) server_negotiation.cc:407] Connection header received
0926 06:15:15.938344 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:15.938346 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:15.938360 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:15.938434 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:15.939016 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:15.939747 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:15.940520 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:15.940873 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:15.940948 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:15.943433 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:15.943442 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:15.943442 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:15.943449 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:15.944985 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:15.945431 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:15.945434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:15.945434 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:15.945494 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:15.945752 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:15.945753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:15.945753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:15.945905 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:15.946049 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:15.946119 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 06:15:15.946191 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":95,"threads_started":1}
Sep 26 06:15:15 dist-test-slave-hndd krb5kdc[6023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:15:15.955338 6108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:15.947631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46118 (local address 127.2.110.254:34755)
0926 06:15:15.947700 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:15.947702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:15.947712 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:15:15.947805 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:15.947806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:15.947815 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:15.947872 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:15.948359 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:15.948848 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:15.949585 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:15.949775 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:15.949856 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:15.952237 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:15.952238 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:15.952239 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:15.952248 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:15.954035 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:15.954425 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:15.954426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:15.954427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:15.954486 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:15.954704 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:15.954705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:15.954705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:15.954873 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:15.954983 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:15.955127 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 06:15:15.955209 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260926 06:15:15.955626 6068 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46118
I20260926 06:15:15.956032 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 6130
I20260926 06:15:15.962764 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 6264
I20260926 06:15:15.967618 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 6398
I20260926 06:15:15.972457 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 6036
2026-09-26T06:15:15Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1958 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:15:16 dist-test-slave-hndd krb5kdc[6539](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:15:16 dist-test-slave-hndd krb5kdc[6539](info): set up 2 sockets
Sep 26 06:15:16 dist-test-slave-hndd krb5kdc[6539](info): commencing operation
krb5kdc: starting...
W20260926 06:15:17.029613 2491 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Sep 26 06:15:17 dist-test-slave-hndd krb5kdc[6539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:15:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:15:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
I20260926 06:15:17.157110 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:35383
--webserver_interface=127.2.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:40387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.110.254:35383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:15:17.239100 6552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:17.239385 6552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:17.239532 6552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:17.240969 6552 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:15:17.241053 6552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:17.241086 6552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:17.241117 6552 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:15:17.241151 6552 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:15:17.243014 6552 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:40387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.110.254:35383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:35383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:17.243348 6552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:17.243611 6552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:17.249253 6560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:17.249260 6558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:17.249259 6557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:17.249552 6552 server_base.cc:1034] running on GCE node
I20260926 06:15:17.249858 6552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:17.250164 6552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:17.251325 6552 hybrid_clock.cc:648] HybridClock initialized: now 1790403317251303 us; error 41 us; skew 500 ppm
Sep 26 06:15:17 dist-test-slave-hndd krb5kdc[6539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403317, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:17.253942 6552 init.cc:377] Logged in from keytab as kudu/127.2.110.254@KRBTEST.COM (short username kudu)
I20260926 06:15:17.254330 6552 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:17.254381 6552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:17.254473 6552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:17.255479 6552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/master-0/data/instance:
uuid: "311ed02438bf4bcbbde1639460f7c171"
format_stamp: "Formatted at 2026-09-26 06:15:17 on dist-test-slave-hndd"
server_key: "41f57c2a7efd304ee003d6cc676a3988"
server_key_iv: "c1d43d23e4282fa44bfa4d3b771ffb06"
server_key_version: "encryptionkey@0"
I20260926 06:15:17.255775 6552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/master-0/wal/instance:
uuid: "311ed02438bf4bcbbde1639460f7c171"
format_stamp: "Formatted at 2026-09-26 06:15:17 on dist-test-slave-hndd"
server_key: "41f57c2a7efd304ee003d6cc676a3988"
server_key_iv: "c1d43d23e4282fa44bfa4d3b771ffb06"
server_key_version: "encryptionkey@0"
I20260926 06:15:17.257149 6552 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 06:15:17.258121 6566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:17.258368 6552 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:17.258476 6552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "311ed02438bf4bcbbde1639460f7c171"
format_stamp: "Formatted at 2026-09-26 06:15:17 on dist-test-slave-hndd"
server_key: "41f57c2a7efd304ee003d6cc676a3988"
server_key_iv: "c1d43d23e4282fa44bfa4d3b771ffb06"
server_key_version: "encryptionkey@0"
I20260926 06:15:17.258566 6552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:17.273093 6552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:17.275512 6552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:17.275674 6552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:17.279811 6552 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.254:35383
I20260926 06:15:17.279840 6618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.254:35383 every 8 connection(s)
I20260926 06:15:17.280150 6552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:15:17.281080 6619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:17.282519 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 6552
I20260926 06:15:17.282635 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/master-0/wal/instance
I20260926 06:15:17.282814 2491 external_mini_cluster.cc:1428] Setting key 6bdf560054d71a64ca29fce64d4013a2
I20260926 06:15:17.283947 6619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 311ed02438bf4bcbbde1639460f7c171: Bootstrap starting.
I20260926 06:15:17.284719 6619 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 311ed02438bf4bcbbde1639460f7c171: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:17.285095 6619 log.cc:824] T 00000000000000000000000000000000 P 311ed02438bf4bcbbde1639460f7c171: Log is configured to *not* fsync() on all Append() calls
I20260926 06:15:17.285864 6619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 311ed02438bf4bcbbde1639460f7c171: No bootstrap required, opened a new log
I20260926 06:15:17.288242 6619 raft_consensus.cc:348] T 00000000000000000000000000000000 P 311ed02438bf4bcbbde1639460f7c171 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "311ed02438bf4bcbbde1639460f7c171" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 35383 } }
I20260926 06:15:17.288437 6619 raft_consensus.cc:374] T 00000000000000000000000000000000 P 311ed02438bf4bcbbde1639460f7c171 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:17.288480 6619 raft_consensus.cc:729] T 00000000000000000000000000000000 P 311ed02438bf4bcbbde1639460f7c171 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 311ed02438bf4bcbbde1639460f7c171, State: Initialized, Role: FOLLOWER
I20260926 06:15:17.288625 6619 consensus_queue.cc:260] T 00000000000000000000000000000000 P 311ed02438bf4bcbbde1639460f7c171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "311ed02438bf4bcbbde1639460f7c171" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 35383 } }
I20260926 06:15:17.288712 6619 raft_consensus.cc:388] T 00000000000000000000000000000000 P 311ed02438bf4bcbbde1639460f7c171 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:15:17.288748 6619 raft_consensus.cc:482] T 00000000000000000000000000000000 P 311ed02438bf4bcbbde1639460f7c171 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:15:17.288805 6619 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 311ed02438bf4bcbbde1639460f7c171 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:17.289637 6619 raft_consensus.cc:504] T 00000000000000000000000000000000 P 311ed02438bf4bcbbde1639460f7c171 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "311ed02438bf4bcbbde1639460f7c171" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 35383 } }
I20260926 06:15:17.289765 6619 leader_election.cc:302] T 00000000000000000000000000000000 P 311ed02438bf4bcbbde1639460f7c171 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 311ed02438bf4bcbbde1639460f7c171; no voters:
Sep 26 06:15:17 dist-test-slave-hndd krb5kdc[6539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:15:17.289990 6619 leader_election.cc:288] T 00000000000000000000000000000000 P 311ed02438bf4bcbbde1639460f7c171 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:15:17.290010 6624 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 311ed02438bf4bcbbde1639460f7c171 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:15:17.290129 6624 raft_consensus.cc:686] T 00000000000000000000000000000000 P 311ed02438bf4bcbbde1639460f7c171 [term 1 LEADER]: Becoming Leader. State: Replica: 311ed02438bf4bcbbde1639460f7c171, State: Running, Role: LEADER
I20260926 06:15:17.290262 6619 sys_catalog.cc:564] T 00000000000000000000000000000000 P 311ed02438bf4bcbbde1639460f7c171 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:15:17.290285 6624 consensus_queue.cc:237] T 00000000000000000000000000000000 P 311ed02438bf4bcbbde1639460f7c171 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "311ed02438bf4bcbbde1639460f7c171" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 35383 } }
I20260926 06:15:17.290702 6625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 311ed02438bf4bcbbde1639460f7c171 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "311ed02438bf4bcbbde1639460f7c171" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "311ed02438bf4bcbbde1639460f7c171" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 35383 } } }
I20260926 06:15:17.290781 6625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 311ed02438bf4bcbbde1639460f7c171 [sys.catalog]: This master's current role is: LEADER
I20260926 06:15:17.290978 6626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 311ed02438bf4bcbbde1639460f7c171 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 311ed02438bf4bcbbde1639460f7c171. Latest consensus state: current_term: 1 leader_uuid: "311ed02438bf4bcbbde1639460f7c171" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "311ed02438bf4bcbbde1639460f7c171" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 35383 } } }
I20260926 06:15:17.291127 6626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 311ed02438bf4bcbbde1639460f7c171 [sys.catalog]: This master's current role is: LEADER
I20260926 06:15:17.291792 6630 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:15:17.292335 6630 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:15:17.294732 6622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:17.283379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50252 (local address 127.2.110.254:35383)
0926 06:15:17.283557 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:17.283570 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:17.283600 (+ 30us) server_negotiation.cc:407] Connection header received
0926 06:15:17.283999 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:17.284005 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:17.284049 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:17.284219 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:17.285040 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:17.286176 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:17.287253 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:17.287676 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:17.287768 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:17.290605 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:17.290608 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:17.290609 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:17.290642 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:17.292904 (+ 2262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:17.293367 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:17.293371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:17.293371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:17.293478 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:17.293708 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:17.293709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:17.293710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:17.294026 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:17.294171 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:17.294353 (+ 182us) server_negotiation.cc:299] Negotiation successful
0926 06:15:17.294505 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":92,"threads_started":1}
I20260926 06:15:17.295001 6630 catalog_manager.cc:1347] Generated new cluster ID: e78aaf2fb2ae4c43a5d9c98d743b7aec
I20260926 06:15:17.295056 6630 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:15:17.312390 6630 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:15:17.313177 6630 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:15:17.318293 6630 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 311ed02438bf4bcbbde1639460f7c171: Generated new TSK 0
I20260926 06:15:17.318516 6630 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab.
Entry for principal kudu/127.2.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab.
I20260926 06:15:17.355213 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.193:0
--local_ip_for_outbound_sockets=127.2.110.193
--webserver_interface=127.2.110.193
--webserver_port=0
--tserver_master_addrs=127.2.110.254:35383
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab
--principal=kudu/127.2.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:40387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:15:17.443583 6645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:17.443813 6645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:17.443840 6645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:17.444983 6645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:17.445022 6645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:17.445056 6645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.193
I20260926 06:15:17.446686 6645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:40387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab
--principal=kudu/127.2.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.110.193
--webserver_port=0
--tserver_master_addrs=127.2.110.254:35383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.193
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:17.446936 6645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:17.447175 6645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:17.452407 6653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:17.452517 6645 server_base.cc:1034] running on GCE node
W20260926 06:15:17.452414 6651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:17.452411 6650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:17.452854 6645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:17.453104 6645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:17.454293 6645 hybrid_clock.cc:648] HybridClock initialized: now 1790403317454269 us; error 65 us; skew 500 ppm
Sep 26 06:15:17 dist-test-slave-hndd krb5kdc[6539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403317, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:17.457118 6645 init.cc:377] Logged in from keytab as kudu/127.2.110.193@KRBTEST.COM (short username kudu)
I20260926 06:15:17.457556 6645 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:17.457633 6645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:17.457779 6645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:17.458843 6645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/ts-0/data/instance:
uuid: "5fa61e44a4254111b0c894699426d6be"
format_stamp: "Formatted at 2026-09-26 06:15:17 on dist-test-slave-hndd"
server_key: "d8bc9e2c8957c17005760edaf0f95acb"
server_key_iv: "cec090f8660b62a5f64533d1d4abd4fe"
server_key_version: "encryptionkey@0"
I20260926 06:15:17.459204 6645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/ts-0/wal/instance:
uuid: "5fa61e44a4254111b0c894699426d6be"
format_stamp: "Formatted at 2026-09-26 06:15:17 on dist-test-slave-hndd"
server_key: "d8bc9e2c8957c17005760edaf0f95acb"
server_key_iv: "cec090f8660b62a5f64533d1d4abd4fe"
server_key_version: "encryptionkey@0"
I20260926 06:15:17.460656 6645 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:15:17.461758 6659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:17.461997 6645 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:15:17.462108 6645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/ts-0/wal
uuid: "5fa61e44a4254111b0c894699426d6be"
format_stamp: "Formatted at 2026-09-26 06:15:17 on dist-test-slave-hndd"
server_key: "d8bc9e2c8957c17005760edaf0f95acb"
server_key_iv: "cec090f8660b62a5f64533d1d4abd4fe"
server_key_version: "encryptionkey@0"
I20260926 06:15:17.462204 6645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:17.484766 6645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:17.487558 6645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:17.487689 6645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:17.487915 6645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:15:17.488281 6645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:15:17.488320 6645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:17.488344 6645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:15:17.488358 6645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:17.494966 6645 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.193:35537
I20260926 06:15:17.495023 6772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.193:35537 every 8 connection(s)
I20260926 06:15:17.495326 6645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
I20260926 06:15:17.501302 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 6645
I20260926 06:15:17.501449 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/ts-0/wal/instance
I20260926 06:15:17.501657 2491 external_mini_cluster.cc:1428] Setting key f296b406a37deb5a2f5c24f0dad370e1
Sep 26 06:15:17 dist-test-slave-hndd krb5kdc[6539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403317, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.193@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:15:17.505631 6622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:17.496157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:56461 (local address 127.2.110.254:35383)
0926 06:15:17.496199 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:17.496201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:17.496432 (+ 231us) server_negotiation.cc:407] Connection header received
0926 06:15:17.496968 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:17.496969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:17.496981 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:17.497034 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:17.498067 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:17.498574 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:17.499583 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:17.499790 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:17.499872 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:17.502352 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:17.502353 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:17.502354 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:17.502362 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:17.504128 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:17.504584 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:17.504586 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:17.504586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:17.504648 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:17.504885 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:17.504885 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:17.504886 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:17.505046 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:17.505114 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:17.505392 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 06:15:17.505488 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 06:15:17.505786 6775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:17.496130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35383 (local address 127.2.110.193:56461)
0926 06:15:17.496297 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:17.496332 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:17.496859 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:17.497118 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:17.497122 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:17.497478 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:17.497956 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:17.497961 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:17.498670 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:17.498671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:17.499471 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:17.499473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:17.499874 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:17.499874 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:17.500004 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:17.500537 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:17.500545 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:17.502240 (+ 1695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:17.504224 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:17.504232 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:17.504234 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:17.504500 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:17.504720 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:17.504720 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:17.504720 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:17.504820 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:17.505111 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:17.505115 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:17.505234 (+ 119us) client_negotiation.cc:769] Sending connection context
0926 06:15:17.505319 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 06:15:17.505526 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":74,"threads_started":1}
I20260926 06:15:17.506400 6773 heartbeater.cc:347] Connected to a master server at 127.2.110.254:35383
I20260926 06:15:17.506511 6773 heartbeater.cc:464] Registering TS with master...
I20260926 06:15:17.506695 6773 heartbeater.cc:511] Master 127.2.110.254:35383 requested a full tablet report, sending...
I20260926 06:15:17.507365 6583 ts_manager.cc:194] Registered new tserver with Master: 5fa61e44a4254111b0c894699426d6be (127.2.110.193:35537)
I20260926 06:15:17.508360 6583 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.193@KRBTEST.COM'} at 127.2.110.193:56461
WARNING: no policy specified for kudu/127.2.110.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab.
Entry for principal kudu/127.2.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab.
I20260926 06:15:17.529285 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.194:0
--local_ip_for_outbound_sockets=127.2.110.194
--webserver_interface=127.2.110.194
--webserver_port=0
--tserver_master_addrs=127.2.110.254:35383
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab
--principal=kudu/127.2.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:40387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:15:17.619866 6778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:17.620314 6778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:17.620412 6778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:17.622701 6778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:17.622795 6778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:17.622993 6778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.194
I20260926 06:15:17.626050 6778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:40387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab
--principal=kudu/127.2.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.110.194
--webserver_port=0
--tserver_master_addrs=127.2.110.254:35383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.194
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:17.626389 6778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:17.626713 6778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:17.633095 6786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:17.633205 6783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:17.633164 6784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:17.633412 6778 server_base.cc:1034] running on GCE node
I20260926 06:15:17.633634 6778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:17.633947 6778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:17.635070 6778 hybrid_clock.cc:648] HybridClock initialized: now 1790403317635048 us; error 36 us; skew 500 ppm
Sep 26 06:15:17 dist-test-slave-hndd krb5kdc[6539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403317, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:17.638437 6778 init.cc:377] Logged in from keytab as kudu/127.2.110.194@KRBTEST.COM (short username kudu)
I20260926 06:15:17.638885 6778 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:17.638978 6778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:17.639132 6778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:17.640861 6778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/ts-1/data/instance:
uuid: "86bacaa929434b83925f952dc2f42a02"
format_stamp: "Formatted at 2026-09-26 06:15:17 on dist-test-slave-hndd"
server_key: "47ac70f8ec404b70e70aefb838d77c11"
server_key_iv: "0c67865ab0d5debad9b28b27188cc864"
server_key_version: "encryptionkey@0"
I20260926 06:15:17.641319 6778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/ts-1/wal/instance:
uuid: "86bacaa929434b83925f952dc2f42a02"
format_stamp: "Formatted at 2026-09-26 06:15:17 on dist-test-slave-hndd"
server_key: "47ac70f8ec404b70e70aefb838d77c11"
server_key_iv: "0c67865ab0d5debad9b28b27188cc864"
server_key_version: "encryptionkey@0"
I20260926 06:15:17.642975 6778 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:15:17.643869 6792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:17.644124 6778 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:15:17.644224 6778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/ts-1/wal
uuid: "86bacaa929434b83925f952dc2f42a02"
format_stamp: "Formatted at 2026-09-26 06:15:17 on dist-test-slave-hndd"
server_key: "47ac70f8ec404b70e70aefb838d77c11"
server_key_iv: "0c67865ab0d5debad9b28b27188cc864"
server_key_version: "encryptionkey@0"
I20260926 06:15:17.644310 6778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:17.659864 6778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:17.662500 6778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:17.662676 6778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:17.662917 6778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:15:17.663242 6778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:15:17.663301 6778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:17.663347 6778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:15:17.663370 6778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:17.670480 6778 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.194:39099
I20260926 06:15:17.670516 6905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.194:39099 every 8 connection(s)
I20260926 06:15:17.670837 6778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
I20260926 06:15:17.674631 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 6778
I20260926 06:15:17.674743 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/ts-1/wal/instance
I20260926 06:15:17.674974 2491 external_mini_cluster.cc:1428] Setting key 6d865ad2c66a615acd20c59212fd563b
Sep 26 06:15:17 dist-test-slave-hndd krb5kdc[6539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403317, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.194@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:15:17.683660 6622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:17.671684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:58203 (local address 127.2.110.254:35383)
0926 06:15:17.671762 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:17.671764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:17.672034 (+ 270us) server_negotiation.cc:407] Connection header received
0926 06:15:17.672776 (+ 742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:17.672777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:17.672794 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:17.672871 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:17.674065 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:17.674708 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:17.676273 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:17.676482 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:17.676551 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:17.680202 (+ 3651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:17.680204 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:17.680204 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:17.680214 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:17.682126 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:17.682596 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:17.682597 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:17.682598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:17.682666 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:17.682926 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:17.682926 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:17.682927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:17.683086 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:17.683160 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:17.683430 (+ 270us) server_negotiation.cc:299] Negotiation successful
0926 06:15:17.683519 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 06:15:17.683691 6908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:17.671658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35383 (local address 127.2.110.194:58203)
0926 06:15:17.671864 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:17.671925 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:17.672658 (+ 733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:17.672960 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:17.672965 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:17.673430 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:17.673952 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:17.673957 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:17.674856 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:17.674857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:17.676165 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:17.676168 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:17.676581 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:17.676582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:17.676756 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:17.678337 (+ 1581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:17.678347 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:17.680092 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:17.682235 (+ 2143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:17.682242 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:17.682245 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:17.682505 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:17.682740 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:17.682740 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:17.682741 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:17.682852 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:17.683153 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:17.683158 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:17.683287 (+ 129us) client_negotiation.cc:769] Sending connection context
0926 06:15:17.683371 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 06:15:17.683514 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":120,"threads_started":1}
WARNING: no policy specified for kudu/127.2.110.195@KRBTEST.COM; defaulting to no policy
I20260926 06:15:17.684280 6906 heartbeater.cc:347] Connected to a master server at 127.2.110.254:35383
I20260926 06:15:17.684391 6906 heartbeater.cc:464] Registering TS with master...
I20260926 06:15:17.684556 6906 heartbeater.cc:511] Master 127.2.110.254:35383 requested a full tablet report, sending...
I20260926 06:15:17.685003 6583 ts_manager.cc:194] Registered new tserver with Master: 86bacaa929434b83925f952dc2f42a02 (127.2.110.194:39099)
I20260926 06:15:17.685523 6583 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.194@KRBTEST.COM'} at 127.2.110.194:58203
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab.
Entry for principal kudu/127.2.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab.
I20260926 06:15:17.705298 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.195:0
--local_ip_for_outbound_sockets=127.2.110.195
--webserver_interface=127.2.110.195
--webserver_port=0
--tserver_master_addrs=127.2.110.254:35383
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab
--principal=kudu/127.2.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:40387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:15:17.792562 6911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:17.792989 6911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:17.793118 6911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:17.794962 6911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:17.795042 6911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:17.795086 6911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.195
I20260926 06:15:17.796734 6911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:40387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab
--principal=kudu/127.2.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.110.195
--webserver_port=0
--tserver_master_addrs=127.2.110.254:35383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.195
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:17.796964 6911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:17.797180 6911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:17.802619 6916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:17.802613 6919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:17.802613 6917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:17.802644 6911 server_base.cc:1034] running on GCE node
I20260926 06:15:17.803030 6911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:17.803265 6911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:17.804443 6911 hybrid_clock.cc:648] HybridClock initialized: now 1790403317804430 us; error 27 us; skew 500 ppm
Sep 26 06:15:17 dist-test-slave-hndd krb5kdc[6539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403317, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:17.807463 6911 init.cc:377] Logged in from keytab as kudu/127.2.110.195@KRBTEST.COM (short username kudu)
I20260926 06:15:17.807804 6911 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:17.807853 6911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:17.807948 6911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:17.809021 6911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/ts-2/data/instance:
uuid: "0cd64a43dfb3429796bbd59c0db2a343"
format_stamp: "Formatted at 2026-09-26 06:15:17 on dist-test-slave-hndd"
server_key: "9b7fdda69ebe953544210bb40ee8b1bf"
server_key_iv: "45a58cdac667f537e71d217608771509"
server_key_version: "encryptionkey@0"
I20260926 06:15:17.809401 6911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/ts-2/wal/instance:
uuid: "0cd64a43dfb3429796bbd59c0db2a343"
format_stamp: "Formatted at 2026-09-26 06:15:17 on dist-test-slave-hndd"
server_key: "9b7fdda69ebe953544210bb40ee8b1bf"
server_key_iv: "45a58cdac667f537e71d217608771509"
server_key_version: "encryptionkey@0"
I20260926 06:15:17.810742 6911 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:15:17.811937 6925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:17.812270 6911 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:15:17.812366 6911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/ts-2/wal
uuid: "0cd64a43dfb3429796bbd59c0db2a343"
format_stamp: "Formatted at 2026-09-26 06:15:17 on dist-test-slave-hndd"
server_key: "9b7fdda69ebe953544210bb40ee8b1bf"
server_key_iv: "45a58cdac667f537e71d217608771509"
server_key_version: "encryptionkey@0"
I20260926 06:15:17.812433 6911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:17.822368 6911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:17.824819 6911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:17.824944 6911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:17.825160 6911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:15:17.825538 6911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:15:17.825575 6911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:17.825600 6911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:15:17.825645 6911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:17.833118 7038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.195:43377 every 8 connection(s)
I20260926 06:15:17.833314 6911 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.195:43377
I20260926 06:15:17.833710 6911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
Sep 26 06:15:17 dist-test-slave-hndd krb5kdc[6539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403317, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.195@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:15:17.841065 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 6911
I20260926 06:15:17.841198 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/ts-2/wal/instance
I20260926 06:15:17.841450 2491 external_mini_cluster.cc:1428] Setting key b155f78cb494bf1f6e0b219e24c29b95
I20260926 06:15:17.843477 6622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:17.834531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:54793 (local address 127.2.110.254:35383)
0926 06:15:17.834600 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:17.834603 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:17.834788 (+ 185us) server_negotiation.cc:407] Connection header received
0926 06:15:17.835346 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:17.835347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:17.835360 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:17.835426 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:17.836427 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:17.836925 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:17.838046 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:17.838238 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:17.838318 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:17.840566 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:17.840567 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:17.840568 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:17.840577 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:17.842248 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:17.842638 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:17.842639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:17.842640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:17.842696 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:17.842905 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:17.842906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:17.842907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:17.843032 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:17.843100 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:17.843343 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 06:15:17.843412 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 06:15:17.843662 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:17.834441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35383 (local address 127.2.110.195:54793)
0926 06:15:17.834648 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:17.834699 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:17.835221 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:17.835508 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:17.835512 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:17.835805 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:17.836306 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:17.836311 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:17.837035 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:17.837036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:17.837933 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:17.837935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:17.838325 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:17.838326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:17.838457 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:17.839021 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:17.839030 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:17.840453 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:17.842315 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:17.842321 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:17.842323 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:17.842565 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:17.842740 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:17.842741 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:17.842741 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:17.842826 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:17.843103 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:17.843107 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:17.843226 (+ 119us) client_negotiation.cc:769] Sending connection context
0926 06:15:17.843312 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 06:15:17.843438 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":92,"threads_started":1}
I20260926 06:15:17.844174 7039 heartbeater.cc:347] Connected to a master server at 127.2.110.254:35383
I20260926 06:15:17.844264 7039 heartbeater.cc:464] Registering TS with master...
I20260926 06:15:17.844385 7039 heartbeater.cc:511] Master 127.2.110.254:35383 requested a full tablet report, sending...
I20260926 06:15:17.844764 6583 ts_manager.cc:194] Registered new tserver with Master: 0cd64a43dfb3429796bbd59c0db2a343 (127.2.110.195:43377)
I20260926 06:15:17.845115 6583 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.195@KRBTEST.COM'} at 127.2.110.195:54793
I20260926 06:15:17.853361 2491 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 06:15:17.861646 6622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:17.854746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50254 (local address 127.2.110.254:35383)
0926 06:15:17.854824 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:17.854826 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:17.854921 (+ 95us) server_negotiation.cc:407] Connection header received
0926 06:15:17.855027 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:17.855028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:17.855039 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:17.855095 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:17.855673 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:17.856180 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:17.856906 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:17.857121 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:17.857201 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:17.858661 (+ 1460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:17.858662 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:17.858663 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:17.858671 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:17.860226 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:17.860711 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:17.860712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:17.860713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:17.860777 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:17.860989 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:17.860990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:17.860990 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:17.861179 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:17.861299 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:17.861449 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 06:15:17.861539 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 06:15:17.863839 6583 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50254:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:15:17.865585 6583 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:15:17.872731 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:17.868833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51510 (local address 127.2.110.194:39099)
0926 06:15:17.869077 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:17.869081 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:17.869092 (+ 11us) server_negotiation.cc:407] Connection header received
0926 06:15:17.869131 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:17.869133 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:17.869149 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:17.869460 (+ 311us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:15:17.869533 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:17.870204 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:17.871673 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:17.872359 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:17.872449 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:17.872512 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:17.872599 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 06:15:17.872645 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":84,"threads_started":1}
I20260926 06:15:17.872722 7050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:17.868331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.194:39099 (local address 127.0.0.1:51510)
0926 06:15:17.868678 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:17.868687 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:17.868743 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:17.869246 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:17.869247 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:17.869249 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:15:17.869310 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:17.869312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:17.870320 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:17.870322 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:17.871579 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:17.871582 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:17.872459 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:17.872460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:17.872525 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:17.872529 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:15:17.872586 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 06:15:17.872633 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":5,"thread_start_us":95,"threads_started":1}
I20260926 06:15:17.872817 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:17.868402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51002 (local address 127.2.110.195:43377)
0926 06:15:17.868888 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:17.868899 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:17.869024 (+ 125us) server_negotiation.cc:407] Connection header received
0926 06:15:17.869153 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:17.869155 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:17.869174 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:17.869242 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:15:17.869446 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:17.870521 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:17.871503 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:17.872263 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:17.872348 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:17.872413 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:17.872681 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 06:15:17.872736 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":85,"threads_started":1}
I20260926 06:15:17.872855 7051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:17.868331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.193:35537 (local address 127.0.0.1:46176)
0926 06:15:17.868629 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:17.868671 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:17.868743 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:17.869080 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:17.869084 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:17.869086 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:15:17.869161 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:17.869163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:17.870564 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:17.870565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:17.871839 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:17.871840 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:17.872640 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:17.872641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:17.872695 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:17.872699 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:15:17.872746 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 06:15:17.872783 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":90,"threads_started":1}
I20260926 06:15:17.873023 7052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:17.868334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.195:43377 (local address 127.0.0.1:51002)
0926 06:15:17.868961 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:17.868972 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:17.869053 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:17.869321 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:17.869322 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:17.869324 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:15:17.869376 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:17.869378 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:17.870627 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:17.870628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:17.871381 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:17.871383 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:17.872358 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:17.872359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:17.872452 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:17.872468 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 06:15:17.872547 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 06:15:17.872609 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"mutex_wait_us":2,"thread_start_us":105,"threads_started":1}
I20260926 06:15:17.873160 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:17.868638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46176 (local address 127.2.110.193:35537)
0926 06:15:17.868793 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:17.868799 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:17.868812 (+ 13us) server_negotiation.cc:407] Connection header received
0926 06:15:17.868855 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:17.868857 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:17.868874 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:17.868976 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:15:17.869248 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:17.870455 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:17.871923 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:17.872492 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:17.872554 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:17.872606 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:17.873010 (+ 404us) server_negotiation.cc:299] Negotiation successful
0926 06:15:17.873059 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":84,"threads_started":1}
I20260926 06:15:17.873798 6973 tablet_service.cc:1467] Processing CreateTablet for tablet ed8a2b09cf1440efba357e292f1e3774 (DEFAULT_TABLE table=test-table [id=9888e75022894921b3217792f037a1ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:15:17.873847 6707 tablet_service.cc:1467] Processing CreateTablet for tablet ed8a2b09cf1440efba357e292f1e3774 (DEFAULT_TABLE table=test-table [id=9888e75022894921b3217792f037a1ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:15:17.873977 6840 tablet_service.cc:1467] Processing CreateTablet for tablet ed8a2b09cf1440efba357e292f1e3774 (DEFAULT_TABLE table=test-table [id=9888e75022894921b3217792f037a1ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:15:17.874194 6973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed8a2b09cf1440efba357e292f1e3774. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:17.874195 6707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed8a2b09cf1440efba357e292f1e3774. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:17.874257 6840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed8a2b09cf1440efba357e292f1e3774. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:17.876812 7056 tablet_bootstrap.cc:492] T ed8a2b09cf1440efba357e292f1e3774 P 0cd64a43dfb3429796bbd59c0db2a343: Bootstrap starting.
I20260926 06:15:17.877665 7056 tablet_bootstrap.cc:654] T ed8a2b09cf1440efba357e292f1e3774 P 0cd64a43dfb3429796bbd59c0db2a343: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:17.878237 7056 log.cc:824] T ed8a2b09cf1440efba357e292f1e3774 P 0cd64a43dfb3429796bbd59c0db2a343: Log is configured to *not* fsync() on all Append() calls
I20260926 06:15:17.878592 7058 tablet_bootstrap.cc:492] T ed8a2b09cf1440efba357e292f1e3774 P 5fa61e44a4254111b0c894699426d6be: Bootstrap starting.
I20260926 06:15:17.879223 7058 tablet_bootstrap.cc:654] T ed8a2b09cf1440efba357e292f1e3774 P 5fa61e44a4254111b0c894699426d6be: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:17.879400 7056 tablet_bootstrap.cc:492] T ed8a2b09cf1440efba357e292f1e3774 P 0cd64a43dfb3429796bbd59c0db2a343: No bootstrap required, opened a new log
I20260926 06:15:17.879475 7056 ts_tablet_manager.cc:1397] T ed8a2b09cf1440efba357e292f1e3774 P 0cd64a43dfb3429796bbd59c0db2a343: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 06:15:17.879882 7058 log.cc:824] T ed8a2b09cf1440efba357e292f1e3774 P 5fa61e44a4254111b0c894699426d6be: Log is configured to *not* fsync() on all Append() calls
I20260926 06:15:17.880278 7057 tablet_bootstrap.cc:492] T ed8a2b09cf1440efba357e292f1e3774 P 86bacaa929434b83925f952dc2f42a02: Bootstrap starting.
I20260926 06:15:17.880831 7058 tablet_bootstrap.cc:492] T ed8a2b09cf1440efba357e292f1e3774 P 5fa61e44a4254111b0c894699426d6be: No bootstrap required, opened a new log
I20260926 06:15:17.880926 7058 ts_tablet_manager.cc:1397] T ed8a2b09cf1440efba357e292f1e3774 P 5fa61e44a4254111b0c894699426d6be: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:15:17.880946 7057 tablet_bootstrap.cc:654] T ed8a2b09cf1440efba357e292f1e3774 P 86bacaa929434b83925f952dc2f42a02: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:17.881601 7057 log.cc:824] T ed8a2b09cf1440efba357e292f1e3774 P 86bacaa929434b83925f952dc2f42a02: Log is configured to *not* fsync() on all Append() calls
I20260926 06:15:17.882038 7056 raft_consensus.cc:348] T ed8a2b09cf1440efba357e292f1e3774 P 0cd64a43dfb3429796bbd59c0db2a343 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa61e44a4254111b0c894699426d6be" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 35537 } } peers { permanent_uuid: "86bacaa929434b83925f952dc2f42a02" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 39099 } } peers { permanent_uuid: "0cd64a43dfb3429796bbd59c0db2a343" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 43377 } }
I20260926 06:15:17.882249 7056 raft_consensus.cc:374] T ed8a2b09cf1440efba357e292f1e3774 P 0cd64a43dfb3429796bbd59c0db2a343 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:17.882292 7056 raft_consensus.cc:729] T ed8a2b09cf1440efba357e292f1e3774 P 0cd64a43dfb3429796bbd59c0db2a343 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cd64a43dfb3429796bbd59c0db2a343, State: Initialized, Role: FOLLOWER
I20260926 06:15:17.882427 7056 consensus_queue.cc:260] T ed8a2b09cf1440efba357e292f1e3774 P 0cd64a43dfb3429796bbd59c0db2a343 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa61e44a4254111b0c894699426d6be" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 35537 } } peers { permanent_uuid: "86bacaa929434b83925f952dc2f42a02" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 39099 } } peers { permanent_uuid: "0cd64a43dfb3429796bbd59c0db2a343" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 43377 } }
I20260926 06:15:17.882539 7057 tablet_bootstrap.cc:492] T ed8a2b09cf1440efba357e292f1e3774 P 86bacaa929434b83925f952dc2f42a02: No bootstrap required, opened a new log
I20260926 06:15:17.882611 7057 ts_tablet_manager.cc:1397] T ed8a2b09cf1440efba357e292f1e3774 P 86bacaa929434b83925f952dc2f42a02: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 06:15:17.882736 7056 ts_tablet_manager.cc:1428] T ed8a2b09cf1440efba357e292f1e3774 P 0cd64a43dfb3429796bbd59c0db2a343: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260926 06:15:17.882891 7039 heartbeater.cc:503] Master 127.2.110.254:35383 was elected leader, sending a full tablet report...
I20260926 06:15:17.883028 7058 raft_consensus.cc:348] T ed8a2b09cf1440efba357e292f1e3774 P 5fa61e44a4254111b0c894699426d6be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa61e44a4254111b0c894699426d6be" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 35537 } } peers { permanent_uuid: "86bacaa929434b83925f952dc2f42a02" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 39099 } } peers { permanent_uuid: "0cd64a43dfb3429796bbd59c0db2a343" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 43377 } }
I20260926 06:15:17.883193 7058 raft_consensus.cc:374] T ed8a2b09cf1440efba357e292f1e3774 P 5fa61e44a4254111b0c894699426d6be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:17.883229 7058 raft_consensus.cc:729] T ed8a2b09cf1440efba357e292f1e3774 P 5fa61e44a4254111b0c894699426d6be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fa61e44a4254111b0c894699426d6be, State: Initialized, Role: FOLLOWER
I20260926 06:15:17.883349 7058 consensus_queue.cc:260] T ed8a2b09cf1440efba357e292f1e3774 P 5fa61e44a4254111b0c894699426d6be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa61e44a4254111b0c894699426d6be" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 35537 } } peers { permanent_uuid: "86bacaa929434b83925f952dc2f42a02" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 39099 } } peers { permanent_uuid: "0cd64a43dfb3429796bbd59c0db2a343" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 43377 } }
I20260926 06:15:17.883718 7058 ts_tablet_manager.cc:1428] T ed8a2b09cf1440efba357e292f1e3774 P 5fa61e44a4254111b0c894699426d6be: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 06:15:17.883836 6773 heartbeater.cc:503] Master 127.2.110.254:35383 was elected leader, sending a full tablet report...
I20260926 06:15:17.884392 7057 raft_consensus.cc:348] T ed8a2b09cf1440efba357e292f1e3774 P 86bacaa929434b83925f952dc2f42a02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa61e44a4254111b0c894699426d6be" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 35537 } } peers { permanent_uuid: "86bacaa929434b83925f952dc2f42a02" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 39099 } } peers { permanent_uuid: "0cd64a43dfb3429796bbd59c0db2a343" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 43377 } }
I20260926 06:15:17.884534 7057 raft_consensus.cc:374] T ed8a2b09cf1440efba357e292f1e3774 P 86bacaa929434b83925f952dc2f42a02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:17.884588 7057 raft_consensus.cc:729] T ed8a2b09cf1440efba357e292f1e3774 P 86bacaa929434b83925f952dc2f42a02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86bacaa929434b83925f952dc2f42a02, State: Initialized, Role: FOLLOWER
I20260926 06:15:17.884709 7057 consensus_queue.cc:260] T ed8a2b09cf1440efba357e292f1e3774 P 86bacaa929434b83925f952dc2f42a02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa61e44a4254111b0c894699426d6be" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 35537 } } peers { permanent_uuid: "86bacaa929434b83925f952dc2f42a02" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 39099 } } peers { permanent_uuid: "0cd64a43dfb3429796bbd59c0db2a343" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 43377 } }
I20260926 06:15:17.884979 7057 ts_tablet_manager.cc:1428] T ed8a2b09cf1440efba357e292f1e3774 P 86bacaa929434b83925f952dc2f42a02: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 06:15:17.885021 6906 heartbeater.cc:503] Master 127.2.110.254:35383 was elected leader, sending a full tablet report...
W20260926 06:15:17.921494 6907 tablet.cc:2369] T ed8a2b09cf1440efba357e292f1e3774 P 86bacaa929434b83925f952dc2f42a02: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:15:17.922077 7064 raft_consensus.cc:482] T ed8a2b09cf1440efba357e292f1e3774 P 86bacaa929434b83925f952dc2f42a02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:15:17.922168 7064 raft_consensus.cc:504] T ed8a2b09cf1440efba357e292f1e3774 P 86bacaa929434b83925f952dc2f42a02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa61e44a4254111b0c894699426d6be" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 35537 } } peers { permanent_uuid: "86bacaa929434b83925f952dc2f42a02" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 39099 } } peers { permanent_uuid: "0cd64a43dfb3429796bbd59c0db2a343" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 43377 } }
I20260926 06:15:17.922538 7064 leader_election.cc:288] T ed8a2b09cf1440efba357e292f1e3774 P 86bacaa929434b83925f952dc2f42a02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5fa61e44a4254111b0c894699426d6be (127.2.110.193:35537), 0cd64a43dfb3429796bbd59c0db2a343 (127.2.110.195:43377)
I20260926 06:15:17.925559 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:17.922879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:42559 (local address 127.2.110.193:35537)
0926 06:15:17.922984 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:17.922986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:17.922995 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:15:17.923019 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:17.923020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:17.923028 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:17.923109 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:15:17.923276 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:17.923746 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:17.924765 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:17.925157 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:17.925245 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:17.925255 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:17.925454 (+ 199us) server_negotiation.cc:299] Negotiation successful
0926 06:15:17.925494 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 06:15:17.925559 6908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:17.922772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.193:35537 (local address 127.2.110.194:42559)
0926 06:15:17.922834 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:17.922846 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:17.922893 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:17.923116 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:17.923117 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:17.923119 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:15:17.923196 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:17.923198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:17.923835 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:17.923836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:17.924663 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:17.924665 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:17.925244 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:17.925245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:17.925329 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:17.925379 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 06:15:17.925439 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 06:15:17.925484 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":52}
I20260926 06:15:17.925634 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:17.922895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:46991 (local address 127.2.110.195:43377)
0926 06:15:17.922943 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:17.922945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:17.923077 (+ 132us) server_negotiation.cc:407] Connection header received
0926 06:15:17.923141 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:17.923142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:17.923152 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:17.923232 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:15:17.923385 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:17.923838 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:17.924834 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:17.925221 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:17.925310 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:17.925319 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:17.925518 (+ 199us) server_negotiation.cc:299] Negotiation successful
0926 06:15:17.925557 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 06:15:17.925706 7065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:17.922790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.195:43377 (local address 127.2.110.194:46991)
0926 06:15:17.923006 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:17.923015 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:17.923087 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:17.923235 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:17.923236 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:17.923238 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:15:17.923307 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:17.923309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:17.923973 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:17.923974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:17.924737 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:17.924739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:17.925309 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:17.925310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:17.925422 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:17.925431 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 06:15:17.925510 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 06:15:17.925550 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":123,"threads_started":1}
I20260926 06:15:17.925952 6727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed8a2b09cf1440efba357e292f1e3774" candidate_uuid: "86bacaa929434b83925f952dc2f42a02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fa61e44a4254111b0c894699426d6be" is_pre_election: true
I20260926 06:15:17.926106 6727 raft_consensus.cc:2454] T ed8a2b09cf1440efba357e292f1e3774 P 5fa61e44a4254111b0c894699426d6be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86bacaa929434b83925f952dc2f42a02 in term 0.
I20260926 06:15:17.926380 6794 leader_election.cc:302] T ed8a2b09cf1440efba357e292f1e3774 P 86bacaa929434b83925f952dc2f42a02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5fa61e44a4254111b0c894699426d6be, 86bacaa929434b83925f952dc2f42a02; no voters:
I20260926 06:15:17.926347 6993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed8a2b09cf1440efba357e292f1e3774" candidate_uuid: "86bacaa929434b83925f952dc2f42a02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cd64a43dfb3429796bbd59c0db2a343" is_pre_election: true
I20260926 06:15:17.926561 6993 raft_consensus.cc:2454] T ed8a2b09cf1440efba357e292f1e3774 P 0cd64a43dfb3429796bbd59c0db2a343 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86bacaa929434b83925f952dc2f42a02 in term 0.
I20260926 06:15:17.926574 7064 raft_consensus.cc:2781] T ed8a2b09cf1440efba357e292f1e3774 P 86bacaa929434b83925f952dc2f42a02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:15:17.926668 7064 raft_consensus.cc:482] T ed8a2b09cf1440efba357e292f1e3774 P 86bacaa929434b83925f952dc2f42a02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:15:17.926688 7064 raft_consensus.cc:3037] T ed8a2b09cf1440efba357e292f1e3774 P 86bacaa929434b83925f952dc2f42a02 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:17.927436 7064 raft_consensus.cc:504] T ed8a2b09cf1440efba357e292f1e3774 P 86bacaa929434b83925f952dc2f42a02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa61e44a4254111b0c894699426d6be" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 35537 } } peers { permanent_uuid: "86bacaa929434b83925f952dc2f42a02" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 39099 } } peers { permanent_uuid: "0cd64a43dfb3429796bbd59c0db2a343" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 43377 } }
I20260926 06:15:17.927594 7064 leader_election.cc:288] T ed8a2b09cf1440efba357e292f1e3774 P 86bacaa929434b83925f952dc2f42a02 [CANDIDATE]: Term 1 election: Requested vote from peers 5fa61e44a4254111b0c894699426d6be (127.2.110.193:35537), 0cd64a43dfb3429796bbd59c0db2a343 (127.2.110.195:43377)
I20260926 06:15:17.927765 6727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed8a2b09cf1440efba357e292f1e3774" candidate_uuid: "86bacaa929434b83925f952dc2f42a02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fa61e44a4254111b0c894699426d6be"
I20260926 06:15:17.927858 6727 raft_consensus.cc:3037] T ed8a2b09cf1440efba357e292f1e3774 P 5fa61e44a4254111b0c894699426d6be [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:17.927901 6993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed8a2b09cf1440efba357e292f1e3774" candidate_uuid: "86bacaa929434b83925f952dc2f42a02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cd64a43dfb3429796bbd59c0db2a343"
I20260926 06:15:17.928045 6993 raft_consensus.cc:3037] T ed8a2b09cf1440efba357e292f1e3774 P 0cd64a43dfb3429796bbd59c0db2a343 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:17.928686 6727 raft_consensus.cc:2454] T ed8a2b09cf1440efba357e292f1e3774 P 5fa61e44a4254111b0c894699426d6be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86bacaa929434b83925f952dc2f42a02 in term 1.
I20260926 06:15:17.928808 6993 raft_consensus.cc:2454] T ed8a2b09cf1440efba357e292f1e3774 P 0cd64a43dfb3429796bbd59c0db2a343 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86bacaa929434b83925f952dc2f42a02 in term 1.
I20260926 06:15:17.928947 6794 leader_election.cc:302] T ed8a2b09cf1440efba357e292f1e3774 P 86bacaa929434b83925f952dc2f42a02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5fa61e44a4254111b0c894699426d6be, 86bacaa929434b83925f952dc2f42a02; no voters:
I20260926 06:15:17.929068 7064 raft_consensus.cc:2781] T ed8a2b09cf1440efba357e292f1e3774 P 86bacaa929434b83925f952dc2f42a02 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:15:17.929255 7064 raft_consensus.cc:686] T ed8a2b09cf1440efba357e292f1e3774 P 86bacaa929434b83925f952dc2f42a02 [term 1 LEADER]: Becoming Leader. State: Replica: 86bacaa929434b83925f952dc2f42a02, State: Running, Role: LEADER
I20260926 06:15:17.929404 7064 consensus_queue.cc:237] T ed8a2b09cf1440efba357e292f1e3774 P 86bacaa929434b83925f952dc2f42a02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa61e44a4254111b0c894699426d6be" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 35537 } } peers { permanent_uuid: "86bacaa929434b83925f952dc2f42a02" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 39099 } } peers { permanent_uuid: "0cd64a43dfb3429796bbd59c0db2a343" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 43377 } }
I20260926 06:15:17.930255 6583 catalog_manager.cc:5495] T ed8a2b09cf1440efba357e292f1e3774 P 86bacaa929434b83925f952dc2f42a02 reported cstate change: term changed from 0 to 1, leader changed from <none> to 86bacaa929434b83925f952dc2f42a02 (127.2.110.194). New cstate: current_term: 1 leader_uuid: "86bacaa929434b83925f952dc2f42a02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa61e44a4254111b0c894699426d6be" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 35537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86bacaa929434b83925f952dc2f42a02" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 39099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cd64a43dfb3429796bbd59c0db2a343" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 43377 } health_report { overall_health: UNKNOWN } } }
I20260926 06:15:17.951879 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:17.949384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51516 (local address 127.2.110.194:39099)
0926 06:15:17.949433 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:17.949435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:17.949476 (+ 41us) server_negotiation.cc:407] Connection header received
0926 06:15:17.949519 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:17.949520 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:17.949536 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:17.949619 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:15:17.949782 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:17.950256 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:17.951043 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:17.951236 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:17.951316 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:17.951498 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:15:17.951608 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:15:17.951713 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:17.951733 (+ 20us) server_negotiation.cc:299] Negotiation successful
0926 06:15:17.951755 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 06:15:17.954098 6993 raft_consensus.cc:1261] T ed8a2b09cf1440efba357e292f1e3774 P 0cd64a43dfb3429796bbd59c0db2a343 [term 1 FOLLOWER]: Refusing update from remote peer 86bacaa929434b83925f952dc2f42a02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:15:17.954147 6727 raft_consensus.cc:1261] T ed8a2b09cf1440efba357e292f1e3774 P 5fa61e44a4254111b0c894699426d6be [term 1 FOLLOWER]: Refusing update from remote peer 86bacaa929434b83925f952dc2f42a02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:15:17.954595 7064 consensus_queue.cc:1035] T ed8a2b09cf1440efba357e292f1e3774 P 86bacaa929434b83925f952dc2f42a02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cd64a43dfb3429796bbd59c0db2a343" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 43377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:15:17.954768 7066 consensus_queue.cc:1035] T ed8a2b09cf1440efba357e292f1e3774 P 86bacaa929434b83925f952dc2f42a02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fa61e44a4254111b0c894699426d6be" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 35537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:15:17.957279 7069 mvcc.cc:205] Tried to move back new op lower bound from 7333491990336360448 to 7333491990239461376. Current Snapshot: MvccSnapshot[applied={T|T < 7333491990336360448}]
I20260926 06:15:17.958532 7070 mvcc.cc:205] Tried to move back new op lower bound from 7333491990336360448 to 7333491990239461376. Current Snapshot: MvccSnapshot[applied={T|T < 7333491990336360448}]
I20260926 06:15:17.960090 6582 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:15:17.960196 6582 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:15:17.961122 6582 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 311ed02438bf4bcbbde1639460f7c171: Sending DeleteTablet for 3 replicas of tablet ed8a2b09cf1440efba357e292f1e3774
I20260926 06:15:17.961478 6840 tablet_service.cc:1514] Processing DeleteTablet for tablet ed8a2b09cf1440efba357e292f1e3774 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:15:17 UTC) from {username='kudu', principal='kudu/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:51510
I20260926 06:15:17.961514 6707 tablet_service.cc:1514] Processing DeleteTablet for tablet ed8a2b09cf1440efba357e292f1e3774 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:15:17 UTC) from {username='kudu', principal='kudu/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:46176
I20260926 06:15:17.961526 6973 tablet_service.cc:1514] Processing DeleteTablet for tablet ed8a2b09cf1440efba357e292f1e3774 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:15:17 UTC) from {username='kudu', principal='kudu/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:51002
I20260926 06:15:17.961751 7079 tablet_replica.cc:330] T ed8a2b09cf1440efba357e292f1e3774 P 86bacaa929434b83925f952dc2f42a02: stopping tablet replica
I20260926 06:15:17.961854 7079 raft_consensus.cc:2229] T ed8a2b09cf1440efba357e292f1e3774 P 86bacaa929434b83925f952dc2f42a02 [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:15:17.961750 7081 tablet_replica.cc:330] T ed8a2b09cf1440efba357e292f1e3774 P 0cd64a43dfb3429796bbd59c0db2a343: stopping tablet replica
I20260926 06:15:17.961951 7079 raft_consensus.cc:2258] T ed8a2b09cf1440efba357e292f1e3774 P 86bacaa929434b83925f952dc2f42a02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:15:17.962217 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 6645
I20260926 06:15:17.963125 7079 ts_tablet_manager.cc:1905] T ed8a2b09cf1440efba357e292f1e3774 P 86bacaa929434b83925f952dc2f42a02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:15:17.962011 7081 raft_consensus.cc:2229] T ed8a2b09cf1440efba357e292f1e3774 P 0cd64a43dfb3429796bbd59c0db2a343 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:15:17.963320 7081 raft_consensus.cc:2258] T ed8a2b09cf1440efba357e292f1e3774 P 0cd64a43dfb3429796bbd59c0db2a343 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:15:17.963783 7081 ts_tablet_manager.cc:1905] T ed8a2b09cf1440efba357e292f1e3774 P 0cd64a43dfb3429796bbd59c0db2a343: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:15:17.964792 7079 ts_tablet_manager.cc:1918] T ed8a2b09cf1440efba357e292f1e3774 P 86bacaa929434b83925f952dc2f42a02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:15:17.964856 7079 log.cc:1196] T ed8a2b09cf1440efba357e292f1e3774 P 86bacaa929434b83925f952dc2f42a02: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/ts-1/wal/wals/ed8a2b09cf1440efba357e292f1e3774
I20260926 06:15:17.965130 7079 ts_tablet_manager.cc:1939] T ed8a2b09cf1440efba357e292f1e3774 P 86bacaa929434b83925f952dc2f42a02: Deleting consensus metadata
I20260926 06:15:17.966451 7081 ts_tablet_manager.cc:1918] T ed8a2b09cf1440efba357e292f1e3774 P 0cd64a43dfb3429796bbd59c0db2a343: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:15:17.966435 6567 catalog_manager.cc:4841] TS 86bacaa929434b83925f952dc2f42a02 (127.2.110.194:39099): tablet ed8a2b09cf1440efba357e292f1e3774 (table test-table [id=9888e75022894921b3217792f037a1ff]) successfully deleted
I20260926 06:15:17.966524 7081 log.cc:1196] T ed8a2b09cf1440efba357e292f1e3774 P 0cd64a43dfb3429796bbd59c0db2a343: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403292369990-2491-0/minicluster-data/ts-2/wal/wals/ed8a2b09cf1440efba357e292f1e3774
I20260926 06:15:17.967133 7081 ts_tablet_manager.cc:1939] T ed8a2b09cf1440efba357e292f1e3774 P 0cd64a43dfb3429796bbd59c0db2a343: Deleting consensus metadata
I20260926 06:15:17.967969 6570 catalog_manager.cc:4841] TS 0cd64a43dfb3429796bbd59c0db2a343 (127.2.110.195:43377): tablet ed8a2b09cf1440efba357e292f1e3774 (table test-table [id=9888e75022894921b3217792f037a1ff]) successfully deleted
W20260926 06:15:17.969553 6568 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.110.193:35537 (error 108)
I20260926 06:15:17.970165 7052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:17.970029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.193:35537 (local address 127.0.0.1:46190)
0926 06:15:17.970086 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:17.970124 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.193:35537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 06:15:17.970201 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 6778
W20260926 06:15:17.971112 6568 catalog_manager.cc:4584] TS 5fa61e44a4254111b0c894699426d6be (127.2.110.193:35537): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ed8a2b09cf1440efba357e292f1e3774: Network error: Client connection negotiation failed: client connection to 127.2.110.193:35537: connect: Connection refused (error 111)
I20260926 06:15:17.974628 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 6911
I20260926 06:15:17.979483 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 6552
2026-09-26T06:15:17Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2006 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T06:15:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:15:18Z Disabled control of system clock
I20260926 06:15:18.006891 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:42671
--webserver_interface=127.2.110.254
--webserver_port=0
--builtin_ntp_servers=127.2.110.212:37837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.110.254:42671
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 06:15:18.091346 7086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:18.091579 7086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:18.091604 7086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:18.092789 7086 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:15:18.092832 7086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:18.092851 7086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:18.092864 7086 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:15:18.092876 7086 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:15:18.094338 7086 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:37837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.110.254:42671
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:42671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:18.094564 7086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:18.094815 7086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:18.099891 7094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:18.099911 7092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:18.099947 7091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:18.100147 7086 server_base.cc:1034] running on GCE node
I20260926 06:15:18.100440 7086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:18.100754 7086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:18.101904 7086 hybrid_clock.cc:648] HybridClock initialized: now 1790403318101882 us; error 42 us; skew 500 ppm
I20260926 06:15:18.103293 7086 webserver.cc:460] Webserver started at http://127.2.110.254:46293/ using document root <none> and password file <none>
I20260926 06:15:18.103689 7086 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:18.103753 7086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:18.103930 7086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:18.105083 7086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/master-0/data/instance:
uuid: "666e4197c3c24605969906c4ea03d1df"
format_stamp: "Formatted at 2026-09-26 06:15:18 on dist-test-slave-hndd"
server_key: "1de369354e26066ff374c2c424fa7bb6"
server_key_iv: "d3365ec4f90c26adfddb53aad984844f"
server_key_version: "encryptionkey@0"
I20260926 06:15:18.105490 7086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/master-0/wal/instance:
uuid: "666e4197c3c24605969906c4ea03d1df"
format_stamp: "Formatted at 2026-09-26 06:15:18 on dist-test-slave-hndd"
server_key: "1de369354e26066ff374c2c424fa7bb6"
server_key_iv: "d3365ec4f90c26adfddb53aad984844f"
server_key_version: "encryptionkey@0"
I20260926 06:15:18.106950 7086 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:15:18.107841 7100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:18.108186 7086 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:15:18.108289 7086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "666e4197c3c24605969906c4ea03d1df"
format_stamp: "Formatted at 2026-09-26 06:15:18 on dist-test-slave-hndd"
server_key: "1de369354e26066ff374c2c424fa7bb6"
server_key_iv: "d3365ec4f90c26adfddb53aad984844f"
server_key_version: "encryptionkey@0"
I20260926 06:15:18.108371 7086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:18.111920 7086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:18.112206 7086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:18.112357 7086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:18.116343 7086 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.254:42671
I20260926 06:15:18.116389 7152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.254:42671 every 8 connection(s)
I20260926 06:15:18.116775 7086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:15:18.117554 7153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:18.120463 7153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 666e4197c3c24605969906c4ea03d1df: Bootstrap starting.
I20260926 06:15:18.121084 7153 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 666e4197c3c24605969906c4ea03d1df: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:18.121376 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 7086
I20260926 06:15:18.121486 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/master-0/wal/instance
I20260926 06:15:18.121539 7153 log.cc:824] T 00000000000000000000000000000000 P 666e4197c3c24605969906c4ea03d1df: Log is configured to *not* fsync() on all Append() calls
I20260926 06:15:18.121682 2491 external_mini_cluster.cc:1428] Setting key 37c9431f640c2c45d95ee8ee0ed0519c
I20260926 06:15:18.122581 7153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 666e4197c3c24605969906c4ea03d1df: No bootstrap required, opened a new log
I20260926 06:15:18.124473 7157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:18.122324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46154 (local address 127.2.110.254:42671)
0926 06:15:18.122550 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:18.122557 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:18.122581 (+ 24us) server_negotiation.cc:407] Connection header received
0926 06:15:18.123074 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:18.123081 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:18.123119 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:18.123235 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:18.123991 (+ 756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:18.123994 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:18.123995 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:15:18.123999 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:18.124041 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:15:18.124051 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:18.124118 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:18.124189 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 06:15:18.124266 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":106,"threads_started":1}
I20260926 06:15:18.124992 7153 raft_consensus.cc:348] T 00000000000000000000000000000000 P 666e4197c3c24605969906c4ea03d1df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "666e4197c3c24605969906c4ea03d1df" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 42671 } }
I20260926 06:15:18.125183 7153 raft_consensus.cc:374] T 00000000000000000000000000000000 P 666e4197c3c24605969906c4ea03d1df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:18.125231 7153 raft_consensus.cc:729] T 00000000000000000000000000000000 P 666e4197c3c24605969906c4ea03d1df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 666e4197c3c24605969906c4ea03d1df, State: Initialized, Role: FOLLOWER
I20260926 06:15:18.125381 7153 consensus_queue.cc:260] T 00000000000000000000000000000000 P 666e4197c3c24605969906c4ea03d1df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "666e4197c3c24605969906c4ea03d1df" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 42671 } }
I20260926 06:15:18.125502 7153 raft_consensus.cc:388] T 00000000000000000000000000000000 P 666e4197c3c24605969906c4ea03d1df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:15:18.125547 7153 raft_consensus.cc:482] T 00000000000000000000000000000000 P 666e4197c3c24605969906c4ea03d1df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:15:18.125597 7153 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 666e4197c3c24605969906c4ea03d1df [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:18.126507 7153 raft_consensus.cc:504] T 00000000000000000000000000000000 P 666e4197c3c24605969906c4ea03d1df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "666e4197c3c24605969906c4ea03d1df" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 42671 } }
I20260926 06:15:18.126657 7153 leader_election.cc:302] T 00000000000000000000000000000000 P 666e4197c3c24605969906c4ea03d1df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 666e4197c3c24605969906c4ea03d1df; no voters:
I20260926 06:15:18.126842 7153 leader_election.cc:288] T 00000000000000000000000000000000 P 666e4197c3c24605969906c4ea03d1df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:15:18.126930 7158 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 666e4197c3c24605969906c4ea03d1df [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:15:18.127117 7153 sys_catalog.cc:564] T 00000000000000000000000000000000 P 666e4197c3c24605969906c4ea03d1df [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:15:18.127122 7158 raft_consensus.cc:686] T 00000000000000000000000000000000 P 666e4197c3c24605969906c4ea03d1df [term 1 LEADER]: Becoming Leader. State: Replica: 666e4197c3c24605969906c4ea03d1df, State: Running, Role: LEADER
I20260926 06:15:18.127280 7158 consensus_queue.cc:237] T 00000000000000000000000000000000 P 666e4197c3c24605969906c4ea03d1df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "666e4197c3c24605969906c4ea03d1df" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 42671 } }
I20260926 06:15:18.127668 7160 sys_catalog.cc:455] T 00000000000000000000000000000000 P 666e4197c3c24605969906c4ea03d1df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 666e4197c3c24605969906c4ea03d1df. Latest consensus state: current_term: 1 leader_uuid: "666e4197c3c24605969906c4ea03d1df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "666e4197c3c24605969906c4ea03d1df" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 42671 } } }
I20260926 06:15:18.127791 7160 sys_catalog.cc:458] T 00000000000000000000000000000000 P 666e4197c3c24605969906c4ea03d1df [sys.catalog]: This master's current role is: LEADER
I20260926 06:15:18.127658 7159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 666e4197c3c24605969906c4ea03d1df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "666e4197c3c24605969906c4ea03d1df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "666e4197c3c24605969906c4ea03d1df" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 42671 } } }
I20260926 06:15:18.127967 7159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 666e4197c3c24605969906c4ea03d1df [sys.catalog]: This master's current role is: LEADER
I20260926 06:15:18.128278 7167 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:15:18.128700 7167 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:15:18.130285 7167 catalog_manager.cc:1347] Generated new cluster ID: d385117073f745baab31b54f78cc2774
I20260926 06:15:18.130342 7167 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:15:18.146713 7167 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:15:18.146955 7167 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:15:18.151520 7167 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 666e4197c3c24605969906c4ea03d1df: Generated new TSK 0
I20260926 06:15:18.151700 7167 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:15:18.158053 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.193:0
--local_ip_for_outbound_sockets=127.2.110.193
--webserver_interface=127.2.110.193
--webserver_port=0
--tserver_master_addrs=127.2.110.254:42671
--builtin_ntp_servers=127.2.110.212:37837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 06:15:18.243913 7177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:18.244143 7177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:18.244168 7177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:18.245476 7177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:18.245522 7177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:18.245550 7177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.193
I20260926 06:15:18.247020 7177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:37837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.110.193
--webserver_port=0
--tserver_master_addrs=127.2.110.254:42671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.193
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:18.247308 7177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:18.247654 7177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:18.252991 7185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:18.252991 7183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:18.253031 7182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:18.253228 7177 server_base.cc:1034] running on GCE node
I20260926 06:15:18.253510 7177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:18.253824 7177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:18.255018 7177 hybrid_clock.cc:648] HybridClock initialized: now 1790403318254987 us; error 45 us; skew 500 ppm
I20260926 06:15:18.256475 7177 webserver.cc:460] Webserver started at http://127.2.110.193:36097/ using document root <none> and password file <none>
I20260926 06:15:18.256778 7177 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:18.256860 7177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:18.257037 7177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:18.258327 7177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/ts-0/data/instance:
uuid: "e0e2a4785400492496d2d201ec84a7c2"
format_stamp: "Formatted at 2026-09-26 06:15:18 on dist-test-slave-hndd"
server_key: "f315a2b77e129232d7cb93064a0cdb6d"
server_key_iv: "5ed1726c1b1e94de6aae78196957e467"
server_key_version: "encryptionkey@0"
I20260926 06:15:18.258738 7177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/ts-0/wal/instance:
uuid: "e0e2a4785400492496d2d201ec84a7c2"
format_stamp: "Formatted at 2026-09-26 06:15:18 on dist-test-slave-hndd"
server_key: "f315a2b77e129232d7cb93064a0cdb6d"
server_key_iv: "5ed1726c1b1e94de6aae78196957e467"
server_key_version: "encryptionkey@0"
I20260926 06:15:18.260284 7177 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:15:18.261175 7191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:18.261427 7177 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:15:18.261534 7177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/ts-0/wal
uuid: "e0e2a4785400492496d2d201ec84a7c2"
format_stamp: "Formatted at 2026-09-26 06:15:18 on dist-test-slave-hndd"
server_key: "f315a2b77e129232d7cb93064a0cdb6d"
server_key_iv: "5ed1726c1b1e94de6aae78196957e467"
server_key_version: "encryptionkey@0"
I20260926 06:15:18.261624 7177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:18.265112 7177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:18.265486 7177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:18.265645 7177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:18.265864 7177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:15:18.266192 7177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:15:18.266248 7177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:18.266288 7177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:15:18.266316 7177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:18.272207 7177 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.193:32877
I20260926 06:15:18.272279 7304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.193:32877 every 8 connection(s)
I20260926 06:15:18.272652 7177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
I20260926 06:15:18.273182 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 7177
I20260926 06:15:18.273272 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/ts-0/wal/instance
I20260926 06:15:18.273509 2491 external_mini_cluster.cc:1428] Setting key d93f889d5438b818fde1b92c6026f147
I20260926 06:15:18.274586 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.194:0
--local_ip_for_outbound_sockets=127.2.110.194
--webserver_interface=127.2.110.194
--webserver_port=0
--tserver_master_addrs=127.2.110.254:42671
--builtin_ntp_servers=127.2.110.212:37837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 06:15:18.276930 7157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:18.273461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:33555 (local address 127.2.110.254:42671)
0926 06:15:18.273535 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:18.273538 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:18.273865 (+ 327us) server_negotiation.cc:407] Connection header received
0926 06:15:18.274576 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:18.274577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:18.274583 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:18.274626 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:18.276624 (+ 1998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:18.276625 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:18.276626 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:15:18.276632 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:18.276652 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:15:18.276658 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:18.276727 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:18.276816 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 06:15:18.276833 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 06:15:18.277135 7307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:18.273526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42671 (local address 127.2.110.193:33555)
0926 06:15:18.273725 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:18.273774 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:18.274453 (+ 679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:18.274705 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:18.274709 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:18.274721 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:18.276385 (+ 1664us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:15:18.276386 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:18.276443 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:15:18.276490 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:18.276728 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:18.276732 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:18.276734 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 06:15:18.276794 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 06:15:18.276850 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":84,"threads_started":1}
I20260926 06:15:18.277715 7305 heartbeater.cc:347] Connected to a master server at 127.2.110.254:42671
I20260926 06:15:18.277819 7305 heartbeater.cc:464] Registering TS with master...
I20260926 06:15:18.277999 7305 heartbeater.cc:511] Master 127.2.110.254:42671 requested a full tablet report, sending...
I20260926 06:15:18.278522 7117 ts_manager.cc:194] Registered new tserver with Master: e0e2a4785400492496d2d201ec84a7c2 (127.2.110.193:32877)
W20260926 06:15:18.366214 7308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:18.366434 7308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:18.366460 7308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:18.367750 7308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:18.367798 7308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:18.367828 7308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.194
I20260926 06:15:18.369239 7308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:37837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.110.194
--webserver_port=0
--tserver_master_addrs=127.2.110.254:42671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.194
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:18.369570 7308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:18.369890 7308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:18.375415 7313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:18.375442 7314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:18.375571 7308 server_base.cc:1034] running on GCE node
W20260926 06:15:18.375429 7316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:18.376039 7308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:18.376361 7308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:18.377526 7308 hybrid_clock.cc:648] HybridClock initialized: now 1790403318377515 us; error 31 us; skew 500 ppm
I20260926 06:15:18.378855 7308 webserver.cc:460] Webserver started at http://127.2.110.194:43969/ using document root <none> and password file <none>
I20260926 06:15:18.379137 7308 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:18.379209 7308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:18.379362 7308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:18.380438 7308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/ts-1/data/instance:
uuid: "ffb24e0fc3dd4e13bdc1741fce7ea5d4"
format_stamp: "Formatted at 2026-09-26 06:15:18 on dist-test-slave-hndd"
server_key: "5916e7971ab29b1dd13f0ce0a8e9dffb"
server_key_iv: "6798915c0f363343cec8ebfd62f45e20"
server_key_version: "encryptionkey@0"
I20260926 06:15:18.380815 7308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/ts-1/wal/instance:
uuid: "ffb24e0fc3dd4e13bdc1741fce7ea5d4"
format_stamp: "Formatted at 2026-09-26 06:15:18 on dist-test-slave-hndd"
server_key: "5916e7971ab29b1dd13f0ce0a8e9dffb"
server_key_iv: "6798915c0f363343cec8ebfd62f45e20"
server_key_version: "encryptionkey@0"
I20260926 06:15:18.382278 7308 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:15:18.383147 7322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:18.383332 7308 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:15:18.383435 7308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/ts-1/wal
uuid: "ffb24e0fc3dd4e13bdc1741fce7ea5d4"
format_stamp: "Formatted at 2026-09-26 06:15:18 on dist-test-slave-hndd"
server_key: "5916e7971ab29b1dd13f0ce0a8e9dffb"
server_key_iv: "6798915c0f363343cec8ebfd62f45e20"
server_key_version: "encryptionkey@0"
I20260926 06:15:18.383523 7308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:18.387002 7308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:18.387274 7308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:18.387420 7308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:18.387636 7308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:15:18.387950 7308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:15:18.388015 7308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:18.388059 7308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:15:18.388085 7308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:18.393975 7308 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.194:33355
I20260926 06:15:18.394011 7435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.194:33355 every 8 connection(s)
I20260926 06:15:18.394359 7308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
I20260926 06:15:18.397562 7157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:18.395186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:48423 (local address 127.2.110.254:42671)
0926 06:15:18.395252 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:18.395254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:18.395505 (+ 251us) server_negotiation.cc:407] Connection header received
0926 06:15:18.395940 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:18.395941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:18.395947 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:18.396017 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:18.397156 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:18.397157 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:18.397158 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:15:18.397162 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:18.397179 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:15:18.397183 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:18.397246 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:18.397475 (+ 229us) server_negotiation.cc:299] Negotiation successful
0926 06:15:18.397492 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 06:15:18.397778 7438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:18.395168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42671 (local address 127.2.110.194:48423)
0926 06:15:18.395343 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:18.395385 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:18.395848 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:18.396072 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:18.396075 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:18.396083 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:18.396958 (+ 875us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:15:18.396959 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:18.397006 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:15:18.397036 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:18.397276 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:18.397282 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:18.397285 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:15:18.397477 (+ 192us) client_negotiation.cc:240] Negotiation successful
0926 06:15:18.397534 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":79,"threads_started":1}
I20260926 06:15:18.398352 7436 heartbeater.cc:347] Connected to a master server at 127.2.110.254:42671
I20260926 06:15:18.398444 7436 heartbeater.cc:464] Registering TS with master...
I20260926 06:15:18.398592 7436 heartbeater.cc:511] Master 127.2.110.254:42671 requested a full tablet report, sending...
I20260926 06:15:18.399030 7117 ts_manager.cc:194] Registered new tserver with Master: ffb24e0fc3dd4e13bdc1741fce7ea5d4 (127.2.110.194:33355)
I20260926 06:15:18.401494 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 7308
I20260926 06:15:18.401578 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/ts-1/wal/instance
I20260926 06:15:18.401786 2491 external_mini_cluster.cc:1428] Setting key 733ccdbd3098b137fb1526ca82c3f5d1
I20260926 06:15:18.402921 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.195:0
--local_ip_for_outbound_sockets=127.2.110.195
--webserver_interface=127.2.110.195
--webserver_port=0
--tserver_master_addrs=127.2.110.254:42671
--builtin_ntp_servers=127.2.110.212:37837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 06:15:18.488305 7439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:18.488541 7439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:18.488567 7439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:18.490461 7439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:18.490525 7439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:18.490558 7439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.195
I20260926 06:15:18.492177 7439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:37837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.110.195
--webserver_port=0
--tserver_master_addrs=127.2.110.254:42671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.195
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:18.492460 7439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:18.492789 7439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:18.498265 7445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:18.498283 7447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:18.498311 7444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:18.498613 7439 server_base.cc:1034] running on GCE node
I20260926 06:15:18.498924 7439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:18.499222 7439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:18.500368 7439 hybrid_clock.cc:648] HybridClock initialized: now 1790403318500336 us; error 45 us; skew 500 ppm
I20260926 06:15:18.501773 7439 webserver.cc:460] Webserver started at http://127.2.110.195:44657/ using document root <none> and password file <none>
I20260926 06:15:18.502027 7439 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:18.502099 7439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:18.502240 7439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:18.503333 7439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/ts-2/data/instance:
uuid: "0024c75a0dd64f0a9ac743144d51a8fa"
format_stamp: "Formatted at 2026-09-26 06:15:18 on dist-test-slave-hndd"
server_key: "0aac0b2ecc795b17c1c4b52edc29b969"
server_key_iv: "f76687b20f17d8cc8dd252eda0d5ab5b"
server_key_version: "encryptionkey@0"
I20260926 06:15:18.503703 7439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/ts-2/wal/instance:
uuid: "0024c75a0dd64f0a9ac743144d51a8fa"
format_stamp: "Formatted at 2026-09-26 06:15:18 on dist-test-slave-hndd"
server_key: "0aac0b2ecc795b17c1c4b52edc29b969"
server_key_iv: "f76687b20f17d8cc8dd252eda0d5ab5b"
server_key_version: "encryptionkey@0"
I20260926 06:15:18.505079 7439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:15:18.506065 7453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:18.506286 7439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:15:18.506398 7439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/ts-2/wal
uuid: "0024c75a0dd64f0a9ac743144d51a8fa"
format_stamp: "Formatted at 2026-09-26 06:15:18 on dist-test-slave-hndd"
server_key: "0aac0b2ecc795b17c1c4b52edc29b969"
server_key_iv: "f76687b20f17d8cc8dd252eda0d5ab5b"
server_key_version: "encryptionkey@0"
I20260926 06:15:18.506490 7439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:18.509984 7439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:18.510247 7439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:18.510401 7439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:18.510614 7439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:15:18.510924 7439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:15:18.510979 7439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:18.511018 7439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:15:18.511045 7439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:18.516925 7439 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.195:37815
I20260926 06:15:18.517019 7566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.195:37815 every 8 connection(s)
I20260926 06:15:18.517421 7439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
I20260926 06:15:18.518695 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 7439
I20260926 06:15:18.518812 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/ts-2/wal/instance
I20260926 06:15:18.518983 2491 external_mini_cluster.cc:1428] Setting key 20862104e653713debee9f04f6039343
I20260926 06:15:18.520726 7157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:18.518229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:55559 (local address 127.2.110.254:42671)
0926 06:15:18.518285 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:18.518288 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:18.518495 (+ 207us) server_negotiation.cc:407] Connection header received
0926 06:15:18.519151 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:18.519153 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:18.519160 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:18.519219 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:18.520490 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:18.520491 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:18.520491 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:15:18.520496 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:18.520513 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:15:18.520517 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:18.520569 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:18.520638 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 06:15:18.520653 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 06:15:18.520972 7569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:18.518190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42671 (local address 127.2.110.195:55559)
0926 06:15:18.518362 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:18.518403 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:18.518997 (+ 594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:18.519291 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:18.519295 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:18.519306 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:18.520329 (+ 1023us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:15:18.520329 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:18.520359 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:15:18.520387 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:18.520567 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:18.520572 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:18.520575 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:15:18.520650 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 06:15:18.520714 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":67,"threads_started":1}
I20260926 06:15:18.521484 7567 heartbeater.cc:347] Connected to a master server at 127.2.110.254:42671
I20260926 06:15:18.521564 7567 heartbeater.cc:464] Registering TS with master...
I20260926 06:15:18.521677 7567 heartbeater.cc:511] Master 127.2.110.254:42671 requested a full tablet report, sending...
I20260926 06:15:18.522015 7117 ts_manager.cc:194] Registered new tserver with Master: 0024c75a0dd64f0a9ac743144d51a8fa (127.2.110.195:37815)
I20260926 06:15:18.531119 2491 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 06:15:18.533810 7157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:18.532525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46160 (local address 127.2.110.254:42671)
0926 06:15:18.532615 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:18.532618 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:18.532713 (+ 95us) server_negotiation.cc:407] Connection header received
0926 06:15:18.532811 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:18.532812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:18.532817 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:18.532871 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:18.533582 (+ 711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:18.533583 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:18.533584 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:15:18.533589 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:18.533604 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:15:18.533609 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:18.533656 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:18.533712 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 06:15:18.533728 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 06:15:18.535496 7117 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46160:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:15:18.537081 7117 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:15:18.541567 7576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:18.539624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.194:33355 (local address 127.0.0.1:57896)
0926 06:15:18.540164 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:18.540173 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:18.540219 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:18.540423 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:18.540430 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:18.540437 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:18.541045 (+ 608us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:15:18.541045 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:18.541064 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:15:18.541090 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:18.541358 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:18.541365 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:18.541368 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:15:18.541446 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 06:15:18.541463 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":92,"threads_started":1}
I20260926 06:15:18.541838 7578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:18.539796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57896 (local address 127.2.110.194:33355)
0926 06:15:18.540176 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:18.540181 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:18.540228 (+ 47us) server_negotiation.cc:407] Connection header received
0926 06:15:18.540313 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:18.540316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:18.540327 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:18.540405 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:18.541183 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:18.541192 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:18.541192 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:15:18.541197 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:18.541233 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:15:18.541257 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:18.541327 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:18.541462 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 06:15:18.541480 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":71,"threads_started":1}
I20260926 06:15:18.541823 7575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:18.539603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.195:37815 (local address 127.0.0.1:48956)
0926 06:15:18.539962 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:18.540010 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:18.540074 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:18.540905 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:18.540906 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:18.540908 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:18.541427 (+ 519us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:15:18.541428 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:18.541436 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:15:18.541451 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:18.541691 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:18.541692 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:18.541693 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 06:15:18.541736 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 06:15:18.541750 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":106,"threads_started":1}
I20260926 06:15:18.542130 7577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:18.539752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48956 (local address 127.2.110.195:37815)
0926 06:15:18.540259 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:18.540263 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:18.540272 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:15:18.540298 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:18.540300 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:18.540310 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:18.540384 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:18.541541 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:18.541547 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:18.541548 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:15:18.541552 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:18.541584 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:15:18.541611 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:18.541666 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:18.542023 (+ 357us) server_negotiation.cc:299] Negotiation successful
0926 06:15:18.542042 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":111,"threads_started":1}
I20260926 06:15:18.542380 7579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:18.539859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.193:32877 (local address 127.0.0.1:46730)
0926 06:15:18.540776 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:18.540787 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:18.540851 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:18.541368 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:18.541369 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:18.541371 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:18.541928 (+ 557us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:15:18.541928 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:18.541937 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:15:18.541953 (+ 16us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:18.542257 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:18.542258 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:18.542258 (+ 0us) client_negotiation.cc:769] Sending connection context
0926 06:15:18.542302 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 06:15:18.542314 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":897,"thread_start_us":64,"threads_started":1}
I20260926 06:15:18.542634 7580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:18.539959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46730 (local address 127.2.110.193:32877)
0926 06:15:18.540166 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:18.540171 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:18.540914 (+ 743us) server_negotiation.cc:407] Connection header received
0926 06:15:18.540945 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:18.540947 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:18.540978 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:18.541064 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:18.542067 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:18.542074 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:18.542074 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:15:18.542078 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:18.542115 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:15:18.542138 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:18.542228 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:18.542529 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 06:15:18.542547 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":102,"threads_started":1}
I20260926 06:15:18.542562 7370 tablet_service.cc:1467] Processing CreateTablet for tablet 64ef370433fa4859aaf26d46a194eded (DEFAULT_TABLE table=test-table [id=c724828c6f0149a6aa94f608a12e8cff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:15:18.542675 7501 tablet_service.cc:1467] Processing CreateTablet for tablet 64ef370433fa4859aaf26d46a194eded (DEFAULT_TABLE table=test-table [id=c724828c6f0149a6aa94f608a12e8cff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:15:18.542944 7370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64ef370433fa4859aaf26d46a194eded. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:18.542972 7501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64ef370433fa4859aaf26d46a194eded. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:18.543288 7239 tablet_service.cc:1467] Processing CreateTablet for tablet 64ef370433fa4859aaf26d46a194eded (DEFAULT_TABLE table=test-table [id=c724828c6f0149a6aa94f608a12e8cff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:15:18.543602 7239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64ef370433fa4859aaf26d46a194eded. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:18.547830 7583 tablet_bootstrap.cc:492] T 64ef370433fa4859aaf26d46a194eded P e0e2a4785400492496d2d201ec84a7c2: Bootstrap starting.
I20260926 06:15:18.547948 7582 tablet_bootstrap.cc:492] T 64ef370433fa4859aaf26d46a194eded P 0024c75a0dd64f0a9ac743144d51a8fa: Bootstrap starting.
I20260926 06:15:18.548622 7583 tablet_bootstrap.cc:654] T 64ef370433fa4859aaf26d46a194eded P e0e2a4785400492496d2d201ec84a7c2: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:18.548895 7582 tablet_bootstrap.cc:654] T 64ef370433fa4859aaf26d46a194eded P 0024c75a0dd64f0a9ac743144d51a8fa: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:18.549017 7583 log.cc:824] T 64ef370433fa4859aaf26d46a194eded P e0e2a4785400492496d2d201ec84a7c2: Log is configured to *not* fsync() on all Append() calls
I20260926 06:15:18.549543 7581 tablet_bootstrap.cc:492] T 64ef370433fa4859aaf26d46a194eded P ffb24e0fc3dd4e13bdc1741fce7ea5d4: Bootstrap starting.
I20260926 06:15:18.549542 7582 log.cc:824] T 64ef370433fa4859aaf26d46a194eded P 0024c75a0dd64f0a9ac743144d51a8fa: Log is configured to *not* fsync() on all Append() calls
I20260926 06:15:18.549906 7583 tablet_bootstrap.cc:492] T 64ef370433fa4859aaf26d46a194eded P e0e2a4785400492496d2d201ec84a7c2: No bootstrap required, opened a new log
I20260926 06:15:18.549988 7583 ts_tablet_manager.cc:1397] T 64ef370433fa4859aaf26d46a194eded P e0e2a4785400492496d2d201ec84a7c2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:15:18.550271 7581 tablet_bootstrap.cc:654] T 64ef370433fa4859aaf26d46a194eded P ffb24e0fc3dd4e13bdc1741fce7ea5d4: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:18.550464 7582 tablet_bootstrap.cc:492] T 64ef370433fa4859aaf26d46a194eded P 0024c75a0dd64f0a9ac743144d51a8fa: No bootstrap required, opened a new log
I20260926 06:15:18.550542 7582 ts_tablet_manager.cc:1397] T 64ef370433fa4859aaf26d46a194eded P 0024c75a0dd64f0a9ac743144d51a8fa: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 06:15:18.550607 7581 log.cc:824] T 64ef370433fa4859aaf26d46a194eded P ffb24e0fc3dd4e13bdc1741fce7ea5d4: Log is configured to *not* fsync() on all Append() calls
I20260926 06:15:18.551808 7581 tablet_bootstrap.cc:492] T 64ef370433fa4859aaf26d46a194eded P ffb24e0fc3dd4e13bdc1741fce7ea5d4: No bootstrap required, opened a new log
I20260926 06:15:18.551878 7581 ts_tablet_manager.cc:1397] T 64ef370433fa4859aaf26d46a194eded P ffb24e0fc3dd4e13bdc1741fce7ea5d4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:15:18.553011 7583 raft_consensus.cc:348] T 64ef370433fa4859aaf26d46a194eded P e0e2a4785400492496d2d201ec84a7c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0024c75a0dd64f0a9ac743144d51a8fa" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 37815 } } peers { permanent_uuid: "e0e2a4785400492496d2d201ec84a7c2" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 32877 } } peers { permanent_uuid: "ffb24e0fc3dd4e13bdc1741fce7ea5d4" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 33355 } }
I20260926 06:15:18.553193 7583 raft_consensus.cc:374] T 64ef370433fa4859aaf26d46a194eded P e0e2a4785400492496d2d201ec84a7c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:18.553237 7583 raft_consensus.cc:729] T 64ef370433fa4859aaf26d46a194eded P e0e2a4785400492496d2d201ec84a7c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0e2a4785400492496d2d201ec84a7c2, State: Initialized, Role: FOLLOWER
I20260926 06:15:18.553117 7582 raft_consensus.cc:348] T 64ef370433fa4859aaf26d46a194eded P 0024c75a0dd64f0a9ac743144d51a8fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0024c75a0dd64f0a9ac743144d51a8fa" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 37815 } } peers { permanent_uuid: "e0e2a4785400492496d2d201ec84a7c2" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 32877 } } peers { permanent_uuid: "ffb24e0fc3dd4e13bdc1741fce7ea5d4" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 33355 } }
I20260926 06:15:18.553277 7582 raft_consensus.cc:374] T 64ef370433fa4859aaf26d46a194eded P 0024c75a0dd64f0a9ac743144d51a8fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:18.553308 7582 raft_consensus.cc:729] T 64ef370433fa4859aaf26d46a194eded P 0024c75a0dd64f0a9ac743144d51a8fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0024c75a0dd64f0a9ac743144d51a8fa, State: Initialized, Role: FOLLOWER
I20260926 06:15:18.553406 7583 consensus_queue.cc:260] T 64ef370433fa4859aaf26d46a194eded P e0e2a4785400492496d2d201ec84a7c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0024c75a0dd64f0a9ac743144d51a8fa" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 37815 } } peers { permanent_uuid: "e0e2a4785400492496d2d201ec84a7c2" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 32877 } } peers { permanent_uuid: "ffb24e0fc3dd4e13bdc1741fce7ea5d4" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 33355 } }
I20260926 06:15:18.553506 7582 consensus_queue.cc:260] T 64ef370433fa4859aaf26d46a194eded P 0024c75a0dd64f0a9ac743144d51a8fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0024c75a0dd64f0a9ac743144d51a8fa" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 37815 } } peers { permanent_uuid: "e0e2a4785400492496d2d201ec84a7c2" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 32877 } } peers { permanent_uuid: "ffb24e0fc3dd4e13bdc1741fce7ea5d4" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 33355 } }
I20260926 06:15:18.553697 7583 ts_tablet_manager.cc:1428] T 64ef370433fa4859aaf26d46a194eded P e0e2a4785400492496d2d201ec84a7c2: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260926 06:15:18.553777 7582 ts_tablet_manager.cc:1428] T 64ef370433fa4859aaf26d46a194eded P 0024c75a0dd64f0a9ac743144d51a8fa: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260926 06:15:18.553831 7305 heartbeater.cc:503] Master 127.2.110.254:42671 was elected leader, sending a full tablet report...
I20260926 06:15:18.553923 7567 heartbeater.cc:503] Master 127.2.110.254:42671 was elected leader, sending a full tablet report...
I20260926 06:15:18.554689 7581 raft_consensus.cc:348] T 64ef370433fa4859aaf26d46a194eded P ffb24e0fc3dd4e13bdc1741fce7ea5d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0024c75a0dd64f0a9ac743144d51a8fa" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 37815 } } peers { permanent_uuid: "e0e2a4785400492496d2d201ec84a7c2" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 32877 } } peers { permanent_uuid: "ffb24e0fc3dd4e13bdc1741fce7ea5d4" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 33355 } }
I20260926 06:15:18.554869 7581 raft_consensus.cc:374] T 64ef370433fa4859aaf26d46a194eded P ffb24e0fc3dd4e13bdc1741fce7ea5d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:18.554914 7581 raft_consensus.cc:729] T 64ef370433fa4859aaf26d46a194eded P ffb24e0fc3dd4e13bdc1741fce7ea5d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffb24e0fc3dd4e13bdc1741fce7ea5d4, State: Initialized, Role: FOLLOWER
I20260926 06:15:18.555047 7581 consensus_queue.cc:260] T 64ef370433fa4859aaf26d46a194eded P ffb24e0fc3dd4e13bdc1741fce7ea5d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0024c75a0dd64f0a9ac743144d51a8fa" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 37815 } } peers { permanent_uuid: "e0e2a4785400492496d2d201ec84a7c2" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 32877 } } peers { permanent_uuid: "ffb24e0fc3dd4e13bdc1741fce7ea5d4" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 33355 } }
I20260926 06:15:18.555327 7581 ts_tablet_manager.cc:1428] T 64ef370433fa4859aaf26d46a194eded P ffb24e0fc3dd4e13bdc1741fce7ea5d4: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260926 06:15:18.555473 7436 heartbeater.cc:503] Master 127.2.110.254:42671 was elected leader, sending a full tablet report...
W20260926 06:15:18.644979 7437 tablet.cc:2369] T 64ef370433fa4859aaf26d46a194eded P ffb24e0fc3dd4e13bdc1741fce7ea5d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:15:18.654675 7589 raft_consensus.cc:482] T 64ef370433fa4859aaf26d46a194eded P ffb24e0fc3dd4e13bdc1741fce7ea5d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:15:18.654806 7589 raft_consensus.cc:504] T 64ef370433fa4859aaf26d46a194eded P ffb24e0fc3dd4e13bdc1741fce7ea5d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0024c75a0dd64f0a9ac743144d51a8fa" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 37815 } } peers { permanent_uuid: "e0e2a4785400492496d2d201ec84a7c2" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 32877 } } peers { permanent_uuid: "ffb24e0fc3dd4e13bdc1741fce7ea5d4" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 33355 } }
I20260926 06:15:18.655161 7589 leader_election.cc:288] T 64ef370433fa4859aaf26d46a194eded P ffb24e0fc3dd4e13bdc1741fce7ea5d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0024c75a0dd64f0a9ac743144d51a8fa (127.2.110.195:37815), e0e2a4785400492496d2d201ec84a7c2 (127.2.110.193:32877)
I20260926 06:15:18.656644 7438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:18.655354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.195:37815 (local address 127.2.110.194:54603)
0926 06:15:18.655420 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:18.655434 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:18.655477 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:18.655708 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:18.655710 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:18.655713 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:18.656245 (+ 532us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:15:18.656245 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:18.656255 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:15:18.656270 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:18.656477 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:18.656478 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:18.656479 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 06:15:18.656557 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 06:15:18.656575 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 06:15:18.656644 7577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:18.655473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:54603 (local address 127.2.110.195:37815)
0926 06:15:18.655539 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:18.655541 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:18.655552 (+ 11us) server_negotiation.cc:407] Connection header received
0926 06:15:18.655584 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:18.655585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:18.655591 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:18.655699 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:18.656367 (+ 668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:18.656368 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:18.656369 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:15:18.656373 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:18.656391 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:15:18.656398 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:18.656465 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:18.656564 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 06:15:18.656578 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 06:15:18.656765 7580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:18.655663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:45163 (local address 127.2.110.193:32877)
0926 06:15:18.655725 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:18.655727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:18.655781 (+ 54us) server_negotiation.cc:407] Connection header received
0926 06:15:18.655843 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:18.655844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:18.655849 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:18.655906 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:18.656524 (+ 618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:18.656526 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:18.656526 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:15:18.656538 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:18.656552 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:15:18.656556 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:18.656632 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:18.656688 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 06:15:18.656702 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 06:15:18.656782 7590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:18.655523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.193:32877 (local address 127.2.110.194:45163)
0926 06:15:18.655728 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:18.655738 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:18.655790 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:18.655912 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:18.655913 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:18.655916 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:18.656443 (+ 527us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:15:18.656444 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:18.656451 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:15:18.656464 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:18.656628 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:18.656629 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:18.656629 (+ 0us) client_negotiation.cc:769] Sending connection context
0926 06:15:18.656687 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 06:15:18.656702 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":123,"threads_started":1}
I20260926 06:15:18.657110 7259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ef370433fa4859aaf26d46a194eded" candidate_uuid: "ffb24e0fc3dd4e13bdc1741fce7ea5d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0e2a4785400492496d2d201ec84a7c2" is_pre_election: true
I20260926 06:15:18.657274 7259 raft_consensus.cc:2454] T 64ef370433fa4859aaf26d46a194eded P e0e2a4785400492496d2d201ec84a7c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffb24e0fc3dd4e13bdc1741fce7ea5d4 in term 0.
I20260926 06:15:18.657213 7521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ef370433fa4859aaf26d46a194eded" candidate_uuid: "ffb24e0fc3dd4e13bdc1741fce7ea5d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0024c75a0dd64f0a9ac743144d51a8fa" is_pre_election: true
I20260926 06:15:18.657408 7521 raft_consensus.cc:2454] T 64ef370433fa4859aaf26d46a194eded P 0024c75a0dd64f0a9ac743144d51a8fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffb24e0fc3dd4e13bdc1741fce7ea5d4 in term 0.
I20260926 06:15:18.657672 7324 leader_election.cc:302] T 64ef370433fa4859aaf26d46a194eded P ffb24e0fc3dd4e13bdc1741fce7ea5d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0024c75a0dd64f0a9ac743144d51a8fa, ffb24e0fc3dd4e13bdc1741fce7ea5d4; no voters:
I20260926 06:15:18.657804 7589 raft_consensus.cc:2781] T 64ef370433fa4859aaf26d46a194eded P ffb24e0fc3dd4e13bdc1741fce7ea5d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:15:18.657861 7589 raft_consensus.cc:482] T 64ef370433fa4859aaf26d46a194eded P ffb24e0fc3dd4e13bdc1741fce7ea5d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:15:18.657886 7589 raft_consensus.cc:3037] T 64ef370433fa4859aaf26d46a194eded P ffb24e0fc3dd4e13bdc1741fce7ea5d4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:18.658612 7589 raft_consensus.cc:504] T 64ef370433fa4859aaf26d46a194eded P ffb24e0fc3dd4e13bdc1741fce7ea5d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0024c75a0dd64f0a9ac743144d51a8fa" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 37815 } } peers { permanent_uuid: "e0e2a4785400492496d2d201ec84a7c2" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 32877 } } peers { permanent_uuid: "ffb24e0fc3dd4e13bdc1741fce7ea5d4" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 33355 } }
I20260926 06:15:18.658761 7589 leader_election.cc:288] T 64ef370433fa4859aaf26d46a194eded P ffb24e0fc3dd4e13bdc1741fce7ea5d4 [CANDIDATE]: Term 1 election: Requested vote from peers 0024c75a0dd64f0a9ac743144d51a8fa (127.2.110.195:37815), e0e2a4785400492496d2d201ec84a7c2 (127.2.110.193:32877)
I20260926 06:15:18.658965 7521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ef370433fa4859aaf26d46a194eded" candidate_uuid: "ffb24e0fc3dd4e13bdc1741fce7ea5d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0024c75a0dd64f0a9ac743144d51a8fa"
I20260926 06:15:18.658987 7259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ef370433fa4859aaf26d46a194eded" candidate_uuid: "ffb24e0fc3dd4e13bdc1741fce7ea5d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0e2a4785400492496d2d201ec84a7c2"
I20260926 06:15:18.659058 7521 raft_consensus.cc:3037] T 64ef370433fa4859aaf26d46a194eded P 0024c75a0dd64f0a9ac743144d51a8fa [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:18.659060 7259 raft_consensus.cc:3037] T 64ef370433fa4859aaf26d46a194eded P e0e2a4785400492496d2d201ec84a7c2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:18.659869 7259 raft_consensus.cc:2454] T 64ef370433fa4859aaf26d46a194eded P e0e2a4785400492496d2d201ec84a7c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffb24e0fc3dd4e13bdc1741fce7ea5d4 in term 1.
I20260926 06:15:18.659890 7521 raft_consensus.cc:2454] T 64ef370433fa4859aaf26d46a194eded P 0024c75a0dd64f0a9ac743144d51a8fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffb24e0fc3dd4e13bdc1741fce7ea5d4 in term 1.
I20260926 06:15:18.660053 7324 leader_election.cc:302] T 64ef370433fa4859aaf26d46a194eded P ffb24e0fc3dd4e13bdc1741fce7ea5d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0e2a4785400492496d2d201ec84a7c2, ffb24e0fc3dd4e13bdc1741fce7ea5d4; no voters:
I20260926 06:15:18.660164 7589 raft_consensus.cc:2781] T 64ef370433fa4859aaf26d46a194eded P ffb24e0fc3dd4e13bdc1741fce7ea5d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:15:18.660313 7589 raft_consensus.cc:686] T 64ef370433fa4859aaf26d46a194eded P ffb24e0fc3dd4e13bdc1741fce7ea5d4 [term 1 LEADER]: Becoming Leader. State: Replica: ffb24e0fc3dd4e13bdc1741fce7ea5d4, State: Running, Role: LEADER
I20260926 06:15:18.660426 7589 consensus_queue.cc:237] T 64ef370433fa4859aaf26d46a194eded P ffb24e0fc3dd4e13bdc1741fce7ea5d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0024c75a0dd64f0a9ac743144d51a8fa" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 37815 } } peers { permanent_uuid: "e0e2a4785400492496d2d201ec84a7c2" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 32877 } } peers { permanent_uuid: "ffb24e0fc3dd4e13bdc1741fce7ea5d4" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 33355 } }
I20260926 06:15:18.661255 7116 catalog_manager.cc:5495] T 64ef370433fa4859aaf26d46a194eded P ffb24e0fc3dd4e13bdc1741fce7ea5d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to ffb24e0fc3dd4e13bdc1741fce7ea5d4 (127.2.110.194). New cstate: current_term: 1 leader_uuid: "ffb24e0fc3dd4e13bdc1741fce7ea5d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0024c75a0dd64f0a9ac743144d51a8fa" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 37815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0e2a4785400492496d2d201ec84a7c2" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 32877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ffb24e0fc3dd4e13bdc1741fce7ea5d4" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 33355 } health_report { overall_health: HEALTHY } } }
I20260926 06:15:18.693100 7578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:18.691829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57908 (local address 127.2.110.194:33355)
0926 06:15:18.691912 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:18.691913 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:18.691923 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:15:18.691972 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:18.691973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:18.691987 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:18.692051 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:18.692802 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:18.692803 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:18.692803 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:15:18.692807 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:18.692827 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:15:18.692830 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:18.692868 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:18.692967 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 06:15:18.692980 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 06:15:18.695127 7521 raft_consensus.cc:1261] T 64ef370433fa4859aaf26d46a194eded P 0024c75a0dd64f0a9ac743144d51a8fa [term 1 FOLLOWER]: Refusing update from remote peer ffb24e0fc3dd4e13bdc1741fce7ea5d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:15:18.695127 7259 raft_consensus.cc:1261] T 64ef370433fa4859aaf26d46a194eded P e0e2a4785400492496d2d201ec84a7c2 [term 1 FOLLOWER]: Refusing update from remote peer ffb24e0fc3dd4e13bdc1741fce7ea5d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:15:18.695533 7591 consensus_queue.cc:1035] T 64ef370433fa4859aaf26d46a194eded P ffb24e0fc3dd4e13bdc1741fce7ea5d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0e2a4785400492496d2d201ec84a7c2" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:15:18.695691 7589 consensus_queue.cc:1035] T 64ef370433fa4859aaf26d46a194eded P ffb24e0fc3dd4e13bdc1741fce7ea5d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0024c75a0dd64f0a9ac743144d51a8fa" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:15:18.701380 7117 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:15:18.701529 7117 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:15:18.702529 7117 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 666e4197c3c24605969906c4ea03d1df: Sending DeleteTablet for 3 replicas of tablet 64ef370433fa4859aaf26d46a194eded
I20260926 06:15:18.702983 7370 tablet_service.cc:1514] Processing DeleteTablet for tablet 64ef370433fa4859aaf26d46a194eded with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:15:18 UTC) from {username='slave'} at 127.0.0.1:57896
I20260926 06:15:18.703018 7501 tablet_service.cc:1514] Processing DeleteTablet for tablet 64ef370433fa4859aaf26d46a194eded with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:15:18 UTC) from {username='slave'} at 127.0.0.1:48956
I20260926 06:15:18.703269 7239 tablet_service.cc:1514] Processing DeleteTablet for tablet 64ef370433fa4859aaf26d46a194eded with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:15:18 UTC) from {username='slave'} at 127.0.0.1:46730
I20260926 06:15:18.703517 7606 tablet_replica.cc:330] T 64ef370433fa4859aaf26d46a194eded P e0e2a4785400492496d2d201ec84a7c2: stopping tablet replica
I20260926 06:15:18.703616 7604 tablet_replica.cc:330] T 64ef370433fa4859aaf26d46a194eded P ffb24e0fc3dd4e13bdc1741fce7ea5d4: stopping tablet replica
I20260926 06:15:18.703693 7606 raft_consensus.cc:2229] T 64ef370433fa4859aaf26d46a194eded P e0e2a4785400492496d2d201ec84a7c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:15:18.703703 7604 raft_consensus.cc:2229] T 64ef370433fa4859aaf26d46a194eded P ffb24e0fc3dd4e13bdc1741fce7ea5d4 [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:15:18.703830 7604 raft_consensus.cc:2258] T 64ef370433fa4859aaf26d46a194eded P ffb24e0fc3dd4e13bdc1741fce7ea5d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:15:18.703823 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 7177
I20260926 06:15:18.703830 7606 raft_consensus.cc:2258] T 64ef370433fa4859aaf26d46a194eded P e0e2a4785400492496d2d201ec84a7c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:15:18.704744 7605 tablet_replica.cc:330] T 64ef370433fa4859aaf26d46a194eded P 0024c75a0dd64f0a9ac743144d51a8fa: stopping tablet replica
I20260926 06:15:18.704876 7605 raft_consensus.cc:2229] T 64ef370433fa4859aaf26d46a194eded P 0024c75a0dd64f0a9ac743144d51a8fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:15:18.704998 7605 raft_consensus.cc:2258] T 64ef370433fa4859aaf26d46a194eded P 0024c75a0dd64f0a9ac743144d51a8fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:15:18.705071 7604 ts_tablet_manager.cc:1905] T 64ef370433fa4859aaf26d46a194eded P ffb24e0fc3dd4e13bdc1741fce7ea5d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:15:18.705518 7605 ts_tablet_manager.cc:1905] T 64ef370433fa4859aaf26d46a194eded P 0024c75a0dd64f0a9ac743144d51a8fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:15:18.706967 7604 ts_tablet_manager.cc:1918] T 64ef370433fa4859aaf26d46a194eded P ffb24e0fc3dd4e13bdc1741fce7ea5d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:15:18.707082 7604 log.cc:1196] T 64ef370433fa4859aaf26d46a194eded P ffb24e0fc3dd4e13bdc1741fce7ea5d4: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/ts-1/wal/wals/64ef370433fa4859aaf26d46a194eded
I20260926 06:15:18.707300 7605 ts_tablet_manager.cc:1918] T 64ef370433fa4859aaf26d46a194eded P 0024c75a0dd64f0a9ac743144d51a8fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:15:18.707384 7605 log.cc:1196] T 64ef370433fa4859aaf26d46a194eded P 0024c75a0dd64f0a9ac743144d51a8fa: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403292369990-2491-0/minicluster-data/ts-2/wal/wals/64ef370433fa4859aaf26d46a194eded
I20260926 06:15:18.707420 7604 ts_tablet_manager.cc:1939] T 64ef370433fa4859aaf26d46a194eded P ffb24e0fc3dd4e13bdc1741fce7ea5d4: Deleting consensus metadata
I20260926 06:15:18.707671 7605 ts_tablet_manager.cc:1939] T 64ef370433fa4859aaf26d46a194eded P 0024c75a0dd64f0a9ac743144d51a8fa: Deleting consensus metadata
I20260926 06:15:18.707959 7101 catalog_manager.cc:4841] TS ffb24e0fc3dd4e13bdc1741fce7ea5d4 (127.2.110.194:33355): tablet 64ef370433fa4859aaf26d46a194eded (table test-table [id=c724828c6f0149a6aa94f608a12e8cff]) successfully deleted
I20260926 06:15:18.708047 7102 catalog_manager.cc:4841] TS 0024c75a0dd64f0a9ac743144d51a8fa (127.2.110.195:37815): tablet 64ef370433fa4859aaf26d46a194eded (table test-table [id=c724828c6f0149a6aa94f608a12e8cff]) successfully deleted
W20260926 06:15:18.711622 7102 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.110.193:32877 (error 108)
I20260926 06:15:18.711833 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 7308
I20260926 06:15:18.713174 7579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:18.712991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.193:32877 (local address 127.0.0.1:46740)
0926 06:15:18.713067 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:18.713125 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.193:32877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 06:15:18.713406 7102 catalog_manager.cc:4584] TS e0e2a4785400492496d2d201ec84a7c2 (127.2.110.193:32877): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 64ef370433fa4859aaf26d46a194eded: Network error: Client connection negotiation failed: client connection to 127.2.110.193:32877: connect: Connection refused (error 111)
I20260926 06:15:18.717633 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 7439
I20260926 06:15:18.722594 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 7086
2026-09-26T06:15:18Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (743 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 06:15:18.743441 2491 mini_oidc.cc:152] Starting JWKS server
I20260926 06:15:18.744530 2491 webserver.cc:460] Webserver started at https://127.0.0.1:41027/ using document root <none> and password file <none>
I20260926 06:15:18.744611 2491 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 06:15:18.744776 2491 webserver.cc:460] Webserver started at http://127.0.0.1:44109/ using document root <none> and password file <none>
2026-09-26T06:15:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:15:18Z Disabled control of system clock
I20260926 06:15:18.752885 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:42015
--webserver_interface=127.2.110.254
--webserver_port=0
--builtin_ntp_servers=127.2.110.212:46633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.110.254:42015
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41027/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790403292369990-2491-0/testchainca.cert with env {}
W20260926 06:15:18.838600 7613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:18.838861 7613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:18.838889 7613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:18.840121 7613 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:15:18.840175 7613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:18.840191 7613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:18.840206 7613 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:15:18.840220 7613 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 06:15:18.840235 7613 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 06:15:18.840248 7613 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41027/jwks.json
I20260926 06:15:18.841979 7613 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:46633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790403292369990-2491-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.110.254:42015
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:42015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41027/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.110.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790403292369990-2491-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:18.842217 7613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:18.842424 7613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:18.847695 7619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:18.847827 7618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:18.847878 7613 server_base.cc:1034] running on GCE node
W20260926 06:15:18.847695 7621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:18.848338 7613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:18.848742 7613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:18.849960 7613 hybrid_clock.cc:648] HybridClock initialized: now 1790403318849895 us; error 81 us; skew 500 ppm
I20260926 06:15:18.851455 7613 webserver.cc:460] Webserver started at http://127.2.110.254:44477/ using document root <none> and password file <none>
I20260926 06:15:18.851737 7613 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:18.851814 7613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:18.852005 7613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:18.853150 7613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790403292369990-2491-0/minicluster-data/master-0/data/instance:
uuid: "7b5fbb0c8a6247e99b1c6c679f00b949"
format_stamp: "Formatted at 2026-09-26 06:15:18 on dist-test-slave-hndd"
server_key: "f10daa96d904e59c5f4d9cfab886e3f7"
server_key_iv: "73820cf0451ff476a612ea1c58d47336"
server_key_version: "encryptionkey@0"
I20260926 06:15:18.853610 7613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790403292369990-2491-0/minicluster-data/master-0/wal/instance:
uuid: "7b5fbb0c8a6247e99b1c6c679f00b949"
format_stamp: "Formatted at 2026-09-26 06:15:18 on dist-test-slave-hndd"
server_key: "f10daa96d904e59c5f4d9cfab886e3f7"
server_key_iv: "73820cf0451ff476a612ea1c58d47336"
server_key_version: "encryptionkey@0"
I20260926 06:15:18.855181 7613 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:15:18.856114 7627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:18.856357 7613 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:15:18.856470 7613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "7b5fbb0c8a6247e99b1c6c679f00b949"
format_stamp: "Formatted at 2026-09-26 06:15:18 on dist-test-slave-hndd"
server_key: "f10daa96d904e59c5f4d9cfab886e3f7"
server_key_iv: "73820cf0451ff476a612ea1c58d47336"
server_key_version: "encryptionkey@0"
I20260926 06:15:18.856565 7613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:18.878371 7613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:18.878786 7613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:18.878968 7613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:18.883232 7613 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.254:42015
I20260926 06:15:18.883268 7679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.254:42015 every 8 connection(s)
I20260926 06:15:18.883647 7613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:15:18.884642 7680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:18.887080 7680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b5fbb0c8a6247e99b1c6c679f00b949: Bootstrap starting.
I20260926 06:15:18.887686 7680 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b5fbb0c8a6247e99b1c6c679f00b949: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:18.888020 7680 log.cc:824] T 00000000000000000000000000000000 P 7b5fbb0c8a6247e99b1c6c679f00b949: Log is configured to *not* fsync() on all Append() calls
I20260926 06:15:18.888123 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 7613
I20260926 06:15:18.888232 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790403292369990-2491-0/minicluster-data/master-0/wal/instance
I20260926 06:15:18.888435 2491 external_mini_cluster.cc:1428] Setting key db2780bcf32ecfb67567b6d092acc9dd
I20260926 06:15:18.888981 7680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b5fbb0c8a6247e99b1c6c679f00b949: No bootstrap required, opened a new log
I20260926 06:15:18.891862 7680 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b5fbb0c8a6247e99b1c6c679f00b949 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b5fbb0c8a6247e99b1c6c679f00b949" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 42015 } }
I20260926 06:15:18.892066 7680 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b5fbb0c8a6247e99b1c6c679f00b949 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:18.892097 7680 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b5fbb0c8a6247e99b1c6c679f00b949 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b5fbb0c8a6247e99b1c6c679f00b949, State: Initialized, Role: FOLLOWER
I20260926 06:15:18.892223 7680 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b5fbb0c8a6247e99b1c6c679f00b949 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b5fbb0c8a6247e99b1c6c679f00b949" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 42015 } }
I20260926 06:15:18.892325 7680 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b5fbb0c8a6247e99b1c6c679f00b949 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:15:18.892360 7680 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b5fbb0c8a6247e99b1c6c679f00b949 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:15:18.892410 7680 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b5fbb0c8a6247e99b1c6c679f00b949 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:18.893070 7680 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b5fbb0c8a6247e99b1c6c679f00b949 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b5fbb0c8a6247e99b1c6c679f00b949" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 42015 } }
I20260926 06:15:18.893182 7680 leader_election.cc:302] T 00000000000000000000000000000000 P 7b5fbb0c8a6247e99b1c6c679f00b949 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b5fbb0c8a6247e99b1c6c679f00b949; no voters:
I20260926 06:15:18.893373 7680 leader_election.cc:288] T 00000000000000000000000000000000 P 7b5fbb0c8a6247e99b1c6c679f00b949 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:15:18.893424 7685 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b5fbb0c8a6247e99b1c6c679f00b949 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:15:18.893591 7685 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b5fbb0c8a6247e99b1c6c679f00b949 [term 1 LEADER]: Becoming Leader. State: Replica: 7b5fbb0c8a6247e99b1c6c679f00b949, State: Running, Role: LEADER
I20260926 06:15:18.893652 7680 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b5fbb0c8a6247e99b1c6c679f00b949 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:15:18.893680 7685 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b5fbb0c8a6247e99b1c6c679f00b949 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b5fbb0c8a6247e99b1c6c679f00b949" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 42015 } }
I20260926 06:15:18.894045 7686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b5fbb0c8a6247e99b1c6c679f00b949 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b5fbb0c8a6247e99b1c6c679f00b949" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b5fbb0c8a6247e99b1c6c679f00b949" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 42015 } } }
I20260926 06:15:18.894122 7686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b5fbb0c8a6247e99b1c6c679f00b949 [sys.catalog]: This master's current role is: LEADER
I20260926 06:15:18.894755 7687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b5fbb0c8a6247e99b1c6c679f00b949 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b5fbb0c8a6247e99b1c6c679f00b949. Latest consensus state: current_term: 1 leader_uuid: "7b5fbb0c8a6247e99b1c6c679f00b949" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b5fbb0c8a6247e99b1c6c679f00b949" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 42015 } } }
I20260926 06:15:18.894909 7687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b5fbb0c8a6247e99b1c6c679f00b949 [sys.catalog]: This master's current role is: LEADER
I20260926 06:15:18.894986 7691 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:15:18.895565 7691 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:15:18.896524 7684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:18.888995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57342 (local address 127.2.110.254:42015)
0926 06:15:18.889202 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:18.889211 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:18.889246 (+ 35us) server_negotiation.cc:407] Connection header received
0926 06:15:18.890324 (+ 1078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:18.890334 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:18.890419 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:18.890623 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:18.890785 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:18.892198 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:18.893076 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:18.893778 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:18.894338 (+ 560us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:18.894788 (+ 450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:18.894793 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:18.894793 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:15:18.895026 (+ 233us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:18.895072 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:15:18.895088 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:18.896034 (+ 946us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:18.896084 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 06:15:18.896208 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":84,"threads_started":1}
I20260926 06:15:18.897668 2491 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:15:18.897797 7691 catalog_manager.cc:1347] Generated new cluster ID: aa6384a707e94544bdf435e2a5494d4d
I20260926 06:15:18.897872 7691 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:15:18.916121 7691 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:15:18.916972 7691 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:15:18.922938 7691 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b5fbb0c8a6247e99b1c6c679f00b949: Generated new TSK 0
I20260926 06:15:18.923177 7691 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 06:15:18.953403 7710 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 06:15:18.953547 7710 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 06:15:18.953923 7684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:18.935812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57352 (local address 127.2.110.254:42015)
0926 06:15:18.935864 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:18.935866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:18.935983 (+ 117us) server_negotiation.cc:407] Connection header received
0926 06:15:18.936063 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:18.936064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:18.936077 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:18.936167 (+ 90us) server_negotiation.cc:226] Negotiated authn=JWT
0926 06:15:18.936283 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:18.937096 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:18.937958 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:18.938279 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:18.938357 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:18.938582 (+ 225us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 06:15:18.953461 (+ 14879us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41027/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 06:15:18.953870 (+ 409us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57352: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41027/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":28}
W20260926 06:15:18.954020 7684 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57352: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41027/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 06:15:18.954085 7709 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:18.935729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42015 (local address 127.0.0.1:57352)
0926 06:15:18.935908 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:18.935919 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:18.935983 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:18.936168 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:18.936169 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:18.936170 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0926 06:15:18.936228 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:18.936230 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:18.937207 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:18.937208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:18.937870 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:18.937873 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:18.938367 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:18.938368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:18.938435 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:18.938458 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 06:15:18.953632 (+ 15174us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41027/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 06:15:18.953746 (+ 114us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.110.254:42015: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41027/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":82,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.2.110.254:42015: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41027/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 06:15:18.954527 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 7613
2026-09-26T06:15:18Z chronyd exiting
I20260926 06:15:18.991037 2491 test_util.cc:175] -----------------------------------------------
I20260926 06:15:18.991122 2491 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790403292369990-2491-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (248 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 06:15:18.991945 2491 mini_oidc.cc:152] Starting JWKS server
I20260926 06:15:18.992748 2491 webserver.cc:460] Webserver started at https://127.0.0.1:35391/ using document root <none> and password file <none>
I20260926 06:15:18.992805 2491 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 06:15:18.992976 2491 webserver.cc:460] Webserver started at http://127.0.0.1:43005/ using document root <none> and password file <none>
2026-09-26T06:15:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:15:18Z Disabled control of system clock
I20260926 06:15:19.001672 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:39331
--webserver_interface=127.2.110.254
--webserver_port=0
--builtin_ntp_servers=127.2.110.212:45763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.110.254:39331
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35391/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790403292369990-2491-0/testchainca.cert with env {}
W20260926 06:15:19.089959 7716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:19.090195 7716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:19.090220 7716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:19.091526 7716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:15:19.091588 7716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:19.091601 7716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:19.091614 7716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:15:19.091627 7716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 06:15:19.091640 7716 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 06:15:19.091652 7716 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35391/jwks.json
I20260926 06:15:19.093387 7716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:45763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790403292369990-2491-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.110.254:39331
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:39331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35391/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.110.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790403292369990-2491-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:19.093679 7716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:19.093932 7716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:19.099310 7721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:19.099334 7722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:19.099310 7724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:19.099606 7716 server_base.cc:1034] running on GCE node
I20260926 06:15:19.099864 7716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:19.100179 7716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:19.101405 7716 hybrid_clock.cc:648] HybridClock initialized: now 1790403319101388 us; error 49 us; skew 500 ppm
I20260926 06:15:19.102736 7716 webserver.cc:460] Webserver started at http://127.2.110.254:34775/ using document root <none> and password file <none>
I20260926 06:15:19.103004 7716 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:19.103088 7716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:19.103248 7716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:19.104358 7716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790403292369990-2491-0/minicluster-data/master-0/data/instance:
uuid: "e6ed9dd4d5944a42b13e75d7cf348d62"
format_stamp: "Formatted at 2026-09-26 06:15:19 on dist-test-slave-hndd"
server_key: "f5f27ab4c636c348f3fbed1b847c62d0"
server_key_iv: "2b580eefae7f2935ecace2fb481f44b5"
server_key_version: "encryptionkey@0"
I20260926 06:15:19.104745 7716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790403292369990-2491-0/minicluster-data/master-0/wal/instance:
uuid: "e6ed9dd4d5944a42b13e75d7cf348d62"
format_stamp: "Formatted at 2026-09-26 06:15:19 on dist-test-slave-hndd"
server_key: "f5f27ab4c636c348f3fbed1b847c62d0"
server_key_iv: "2b580eefae7f2935ecace2fb481f44b5"
server_key_version: "encryptionkey@0"
I20260926 06:15:19.106312 7716 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:15:19.107221 7730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:19.107430 7716 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:19.107530 7716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "e6ed9dd4d5944a42b13e75d7cf348d62"
format_stamp: "Formatted at 2026-09-26 06:15:19 on dist-test-slave-hndd"
server_key: "f5f27ab4c636c348f3fbed1b847c62d0"
server_key_iv: "2b580eefae7f2935ecace2fb481f44b5"
server_key_version: "encryptionkey@0"
I20260926 06:15:19.107631 7716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:19.131453 7716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:19.131840 7716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:19.132022 7716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:19.137162 7716 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.254:39331
I20260926 06:15:19.137248 7782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.254:39331 every 8 connection(s)
I20260926 06:15:19.137771 7716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:15:19.138901 7783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:19.141498 7783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6ed9dd4d5944a42b13e75d7cf348d62: Bootstrap starting.
I20260926 06:15:19.142148 7783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6ed9dd4d5944a42b13e75d7cf348d62: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:19.142462 7783 log.cc:824] T 00000000000000000000000000000000 P e6ed9dd4d5944a42b13e75d7cf348d62: Log is configured to *not* fsync() on all Append() calls
I20260926 06:15:19.143271 7783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6ed9dd4d5944a42b13e75d7cf348d62: No bootstrap required, opened a new log
I20260926 06:15:19.144991 7783 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6ed9dd4d5944a42b13e75d7cf348d62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6ed9dd4d5944a42b13e75d7cf348d62" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 39331 } }
I20260926 06:15:19.145150 7783 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6ed9dd4d5944a42b13e75d7cf348d62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:19.145174 7783 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6ed9dd4d5944a42b13e75d7cf348d62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ed9dd4d5944a42b13e75d7cf348d62, State: Initialized, Role: FOLLOWER
I20260926 06:15:19.145411 7783 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6ed9dd4d5944a42b13e75d7cf348d62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6ed9dd4d5944a42b13e75d7cf348d62" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 39331 } }
I20260926 06:15:19.145515 7783 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6ed9dd4d5944a42b13e75d7cf348d62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:15:19.145540 7783 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6ed9dd4d5944a42b13e75d7cf348d62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:15:19.145596 7783 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6ed9dd4d5944a42b13e75d7cf348d62 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:19.146214 7783 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6ed9dd4d5944a42b13e75d7cf348d62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6ed9dd4d5944a42b13e75d7cf348d62" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 39331 } }
I20260926 06:15:19.146308 7783 leader_election.cc:302] T 00000000000000000000000000000000 P e6ed9dd4d5944a42b13e75d7cf348d62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6ed9dd4d5944a42b13e75d7cf348d62; no voters:
I20260926 06:15:19.146500 7783 leader_election.cc:288] T 00000000000000000000000000000000 P e6ed9dd4d5944a42b13e75d7cf348d62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:15:19.146821 7786 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6ed9dd4d5944a42b13e75d7cf348d62 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:15:19.147014 7783 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6ed9dd4d5944a42b13e75d7cf348d62 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:15:19.147295 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 7716
I20260926 06:15:19.147307 7786 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6ed9dd4d5944a42b13e75d7cf348d62 [term 1 LEADER]: Becoming Leader. State: Replica: e6ed9dd4d5944a42b13e75d7cf348d62, State: Running, Role: LEADER
I20260926 06:15:19.147429 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790403292369990-2491-0/minicluster-data/master-0/wal/instance
I20260926 06:15:19.147495 7786 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6ed9dd4d5944a42b13e75d7cf348d62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6ed9dd4d5944a42b13e75d7cf348d62" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 39331 } }
I20260926 06:15:19.148026 2491 external_mini_cluster.cc:1428] Setting key dfd8509eec1ce962d9d1c731ae5648fa
I20260926 06:15:19.149451 7787 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6ed9dd4d5944a42b13e75d7cf348d62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6ed9dd4d5944a42b13e75d7cf348d62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6ed9dd4d5944a42b13e75d7cf348d62" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 39331 } } }
I20260926 06:15:19.149695 7787 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6ed9dd4d5944a42b13e75d7cf348d62 [sys.catalog]: This master's current role is: LEADER
I20260926 06:15:19.150014 7788 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6ed9dd4d5944a42b13e75d7cf348d62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6ed9dd4d5944a42b13e75d7cf348d62. Latest consensus state: current_term: 1 leader_uuid: "e6ed9dd4d5944a42b13e75d7cf348d62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6ed9dd4d5944a42b13e75d7cf348d62" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 39331 } } }
I20260926 06:15:19.150092 7788 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6ed9dd4d5944a42b13e75d7cf348d62 [sys.catalog]: This master's current role is: LEADER
W20260926 06:15:19.150166 7802 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e6ed9dd4d5944a42b13e75d7cf348d62: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 06:15:19.150213 7802 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 06:15:19.150345 7803 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:15:19.150552 7803 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:15:19.152367 7803 catalog_manager.cc:1347] Generated new cluster ID: 40b4ed35f48c43118b6125af07445da3
I20260926 06:15:19.152470 7803 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:15:19.155504 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:19.148948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60816 (local address 127.2.110.254:39331)
0926 06:15:19.149279 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:19.149288 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:19.149326 (+ 38us) server_negotiation.cc:407] Connection header received
0926 06:15:19.149490 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:19.149498 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:19.149594 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:19.150187 (+ 593us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:19.150301 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:19.152004 (+ 1703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:19.153049 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:19.153615 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:19.153768 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:19.154520 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:19.154524 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:19.154524 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:15:19.154718 (+ 194us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:19.154750 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:15:19.154762 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:19.154836 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:19.154912 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 06:15:19.155000 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":104,"threads_started":1}
I20260926 06:15:19.156483 2491 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:15:19.175808 7803 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:15:19.176571 7803 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:15:19.185930 7803 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6ed9dd4d5944a42b13e75d7cf348d62: Generated new TSK 0
I20260926 06:15:19.186162 7803 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 06:15:19.212648 7814 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 06:15:19.212792 7814 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 06:15:19.213117 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:19.195988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60828 (local address 127.2.110.254:39331)
0926 06:15:19.196148 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:19.196152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:19.196171 (+ 19us) server_negotiation.cc:407] Connection header received
0926 06:15:19.196257 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:19.196258 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:19.196277 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:19.196417 (+ 140us) server_negotiation.cc:226] Negotiated authn=JWT
0926 06:15:19.196620 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:19.198161 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:19.199166 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:19.199487 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:19.199568 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:19.199780 (+ 212us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 06:15:19.212811 (+ 13031us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35391/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 06:15:19.213038 (+ 227us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60828: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35391/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":31}
W20260926 06:15:19.213181 7801 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60828: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35391/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260926 06:15:19.213253 7813 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:19.195880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:39331 (local address 127.0.0.1:60828)
0926 06:15:19.196020 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:19.196031 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:19.196092 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:19.196445 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:19.196446 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:19.196449 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0926 06:15:19.196509 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:19.196511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:19.198345 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:19.198347 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:19.199048 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:19.199051 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:19.199589 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:19.199590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:19.199639 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:19.199645 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 06:15:19.213003 (+ 13358us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35391/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 06:15:19.213121 (+ 118us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.110.254:39331: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35391/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":86,"threads_started":1}
I20260926 06:15:19.213855 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 7716
2026-09-26T06:15:19Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (261 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 06:15:19.252961 2491 mini_oidc.cc:152] Starting JWKS server
I20260926 06:15:19.253778 2491 webserver.cc:460] Webserver started at https://127.0.0.1:34127/ using document root <none> and password file <none>
I20260926 06:15:19.253822 2491 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 06:15:19.253947 2491 webserver.cc:460] Webserver started at http://127.0.0.1:38503/ using document root <none> and password file <none>
2026-09-26T06:15:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:15:19Z Disabled control of system clock
I20260926 06:15:19.262852 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:42733
--webserver_interface=127.2.110.254
--webserver_port=0
--builtin_ntp_servers=127.2.110.212:46091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.110.254:42733
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34127/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 06:15:19.345925 7820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:19.346186 7820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:19.346211 7820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:19.348222 7820 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:15:19.348284 7820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:19.348299 7820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:19.348312 7820 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:15:19.348325 7820 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 06:15:19.348340 7820 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 06:15:19.348351 7820 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34127/jwks.json
I20260926 06:15:19.349906 7820 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:46091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790403292369990-2491-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.110.254:42733
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:42733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34127/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:19.350123 7820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:19.350368 7820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:19.355746 7825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:19.355774 7828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:19.355786 7826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:19.356798 7820 server_base.cc:1034] running on GCE node
I20260926 06:15:19.357052 7820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:19.357429 7820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:19.358606 7820 hybrid_clock.cc:648] HybridClock initialized: now 1790403319358585 us; error 43 us; skew 500 ppm
I20260926 06:15:19.359977 7820 webserver.cc:460] Webserver started at http://127.2.110.254:39385/ using document root <none> and password file <none>
I20260926 06:15:19.360244 7820 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:19.360296 7820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:19.360473 7820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:19.361652 7820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790403292369990-2491-0/minicluster-data/master-0/data/instance:
uuid: "8d0528a545554f8aaa987554d71e1bf6"
format_stamp: "Formatted at 2026-09-26 06:15:19 on dist-test-slave-hndd"
server_key: "40249280ed92d0b36878cb8add9da364"
server_key_iv: "c5a4fa0391899ff6eb467ab841d23ae3"
server_key_version: "encryptionkey@0"
I20260926 06:15:19.362026 7820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790403292369990-2491-0/minicluster-data/master-0/wal/instance:
uuid: "8d0528a545554f8aaa987554d71e1bf6"
format_stamp: "Formatted at 2026-09-26 06:15:19 on dist-test-slave-hndd"
server_key: "40249280ed92d0b36878cb8add9da364"
server_key_iv: "c5a4fa0391899ff6eb467ab841d23ae3"
server_key_version: "encryptionkey@0"
I20260926 06:15:19.363547 7820 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:15:19.364477 7834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:19.364728 7820 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:19.364811 7820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "8d0528a545554f8aaa987554d71e1bf6"
format_stamp: "Formatted at 2026-09-26 06:15:19 on dist-test-slave-hndd"
server_key: "40249280ed92d0b36878cb8add9da364"
server_key_iv: "c5a4fa0391899ff6eb467ab841d23ae3"
server_key_version: "encryptionkey@0"
I20260926 06:15:19.364895 7820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:19.373816 7820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:19.374130 7820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:19.374282 7820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:19.380158 7820 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.254:42733
I20260926 06:15:19.380208 7886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.254:42733 every 8 connection(s)
I20260926 06:15:19.380574 7820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:15:19.381301 7887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:19.384053 7887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d0528a545554f8aaa987554d71e1bf6: Bootstrap starting.
I20260926 06:15:19.384689 7887 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d0528a545554f8aaa987554d71e1bf6: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:19.384986 7887 log.cc:824] T 00000000000000000000000000000000 P 8d0528a545554f8aaa987554d71e1bf6: Log is configured to *not* fsync() on all Append() calls
I20260926 06:15:19.385929 7887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d0528a545554f8aaa987554d71e1bf6: No bootstrap required, opened a new log
I20260926 06:15:19.388054 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 7820
I20260926 06:15:19.388188 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790403292369990-2491-0/minicluster-data/master-0/wal/instance
I20260926 06:15:19.388381 2491 external_mini_cluster.cc:1428] Setting key 6a0eb8aac7b8fa994252e1a0f7b7894e
I20260926 06:15:19.389451 7887 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d0528a545554f8aaa987554d71e1bf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0528a545554f8aaa987554d71e1bf6" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 42733 } }
I20260926 06:15:19.389631 7887 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d0528a545554f8aaa987554d71e1bf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:19.389676 7887 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d0528a545554f8aaa987554d71e1bf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d0528a545554f8aaa987554d71e1bf6, State: Initialized, Role: FOLLOWER
I20260926 06:15:19.389882 7887 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d0528a545554f8aaa987554d71e1bf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0528a545554f8aaa987554d71e1bf6" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 42733 } }
I20260926 06:15:19.389985 7887 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d0528a545554f8aaa987554d71e1bf6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:15:19.390026 7887 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d0528a545554f8aaa987554d71e1bf6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:15:19.390084 7887 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d0528a545554f8aaa987554d71e1bf6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:19.390698 7887 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d0528a545554f8aaa987554d71e1bf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0528a545554f8aaa987554d71e1bf6" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 42733 } }
I20260926 06:15:19.390833 7887 leader_election.cc:302] T 00000000000000000000000000000000 P 8d0528a545554f8aaa987554d71e1bf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d0528a545554f8aaa987554d71e1bf6; no voters:
I20260926 06:15:19.391038 7887 leader_election.cc:288] T 00000000000000000000000000000000 P 8d0528a545554f8aaa987554d71e1bf6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:15:19.391067 7892 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d0528a545554f8aaa987554d71e1bf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:15:19.391179 7892 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d0528a545554f8aaa987554d71e1bf6 [term 1 LEADER]: Becoming Leader. State: Replica: 8d0528a545554f8aaa987554d71e1bf6, State: Running, Role: LEADER
I20260926 06:15:19.391319 7887 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d0528a545554f8aaa987554d71e1bf6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:15:19.391297 7892 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d0528a545554f8aaa987554d71e1bf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0528a545554f8aaa987554d71e1bf6" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 42733 } }
I20260926 06:15:19.391675 7893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d0528a545554f8aaa987554d71e1bf6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d0528a545554f8aaa987554d71e1bf6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0528a545554f8aaa987554d71e1bf6" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 42733 } } }
I20260926 06:15:19.391907 7893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d0528a545554f8aaa987554d71e1bf6 [sys.catalog]: This master's current role is: LEADER
I20260926 06:15:19.391841 7894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d0528a545554f8aaa987554d71e1bf6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d0528a545554f8aaa987554d71e1bf6. Latest consensus state: current_term: 1 leader_uuid: "8d0528a545554f8aaa987554d71e1bf6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0528a545554f8aaa987554d71e1bf6" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 42733 } } }
I20260926 06:15:19.392159 7894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d0528a545554f8aaa987554d71e1bf6 [sys.catalog]: This master's current role is: LEADER
I20260926 06:15:19.392529 7899 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:15:19.393085 7899 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:15:19.395342 7899 catalog_manager.cc:1347] Generated new cluster ID: 7e02132d1c544b8b9836d924775f158d
I20260926 06:15:19.395424 7899 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:15:19.395550 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:19.389036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32878 (local address 127.2.110.254:42733)
0926 06:15:19.389447 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:19.389455 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:19.389481 (+ 26us) server_negotiation.cc:407] Connection header received
0926 06:15:19.389579 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:19.389585 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:19.389640 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:19.389812 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:19.390111 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:19.391611 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:19.392924 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:19.393541 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:19.393669 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:19.394781 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:19.394785 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:19.394785 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:15:19.394967 (+ 182us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:19.395001 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:15:19.395011 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:19.395094 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:19.395170 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 06:15:19.395251 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":219,"threads_started":1}
I20260926 06:15:19.396453 2491 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:15:19.403481 7899 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:15:19.404140 7899 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:15:19.409258 7899 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d0528a545554f8aaa987554d71e1bf6: Generated new TSK 0
I20260926 06:15:19.409575 7899 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:15:19.420589 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:19.416600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32880 (local address 127.2.110.254:42733)
0926 06:15:19.416666 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:19.416668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:19.416686 (+ 18us) server_negotiation.cc:407] Connection header received
0926 06:15:19.416722 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:19.416723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:19.416733 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:19.416800 (+ 67us) server_negotiation.cc:226] Negotiated authn=JWT
0926 06:15:19.416980 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:19.417531 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:19.418678 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:19.419079 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:19.419149 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:19.419354 (+ 205us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 06:15:19.420370 (+ 1016us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34127/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 06:15:19.420504 (+ 134us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:32880: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34127/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":40}
W20260926 06:15:19.420657 7891 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:32880: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34127/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260926 06:15:19.420702 7916 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:19.416220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42733 (local address 127.0.0.1:32880)
0926 06:15:19.416398 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:19.416410 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:19.416472 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:19.416821 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:19.416822 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:19.416825 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0926 06:15:19.416902 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:19.416904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:19.417630 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:19.417631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:19.418587 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:19.418590 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:19.419177 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:19.419178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:19.419244 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:19.419250 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 06:15:19.420483 (+ 1233us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34127/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 06:15:19.420593 (+ 110us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.110.254:42733: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34127/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":102,"threads_started":1}
I20260926 06:15:19.420946 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 7820
E20260926 06:15:19.431638 7917 webserver.cc:533] Webserver: sslize failed:
E20260926 06:15:19.431739 7917 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-26T06:15:19Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (205 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T06:15:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:15:19Z Disabled control of system clock
I20260926 06:15:19.466512 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:40043
--webserver_interface=127.2.110.254
--webserver_port=0
--builtin_ntp_servers=127.2.110.212:41965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.110.254:40043
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790403292369990-2491-0/testchainca.cert with env {}
W20260926 06:15:19.552831 7921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:19.553068 7921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:19.553093 7921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:19.554394 7921 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:15:19.554448 7921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:19.554462 7921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:19.554476 7921 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:15:19.554487 7921 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 06:15:19.554502 7921 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 06:15:19.554513 7921 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 06:15:19.556108 7921 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:41965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790403292369990-2491-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.110.254:40043
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:40043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.110.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790403292369990-2491-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:19.556393 7921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:19.556638 7921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:19.561615 7927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:19.561638 7929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:19.561625 7926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:19.561887 7921 server_base.cc:1034] running on GCE node
I20260926 06:15:19.562196 7921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:19.562520 7921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:19.563704 7921 hybrid_clock.cc:648] HybridClock initialized: now 1790403319563696 us; error 35 us; skew 500 ppm
I20260926 06:15:19.565084 7921 webserver.cc:460] Webserver started at http://127.2.110.254:37409/ using document root <none> and password file <none>
I20260926 06:15:19.565415 7921 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:19.565493 7921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:19.565658 7921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:19.566776 7921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790403292369990-2491-0/minicluster-data/master-0/data/instance:
uuid: "e78099f3e87a46d886bb111a7918e647"
format_stamp: "Formatted at 2026-09-26 06:15:19 on dist-test-slave-hndd"
server_key: "186526f2aeb7cecd27843ce6eaf7261f"
server_key_iv: "aea004e76f6d5ebac2b0698ef6680cb6"
server_key_version: "encryptionkey@0"
I20260926 06:15:19.567160 7921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790403292369990-2491-0/minicluster-data/master-0/wal/instance:
uuid: "e78099f3e87a46d886bb111a7918e647"
format_stamp: "Formatted at 2026-09-26 06:15:19 on dist-test-slave-hndd"
server_key: "186526f2aeb7cecd27843ce6eaf7261f"
server_key_iv: "aea004e76f6d5ebac2b0698ef6680cb6"
server_key_version: "encryptionkey@0"
I20260926 06:15:19.568598 7921 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:15:19.569546 7935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:19.569770 7921 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:15:19.569886 7921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "e78099f3e87a46d886bb111a7918e647"
format_stamp: "Formatted at 2026-09-26 06:15:19 on dist-test-slave-hndd"
server_key: "186526f2aeb7cecd27843ce6eaf7261f"
server_key_iv: "aea004e76f6d5ebac2b0698ef6680cb6"
server_key_version: "encryptionkey@0"
I20260926 06:15:19.569981 7921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:19.597302 7921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:19.597970 7921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:19.598186 7921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:19.602523 7921 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.254:40043
I20260926 06:15:19.602560 7987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.254:40043 every 8 connection(s)
I20260926 06:15:19.602963 7921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:15:19.603984 7988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:19.606485 7988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e78099f3e87a46d886bb111a7918e647: Bootstrap starting.
I20260926 06:15:19.607074 7988 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e78099f3e87a46d886bb111a7918e647: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:19.607311 7988 log.cc:824] T 00000000000000000000000000000000 P e78099f3e87a46d886bb111a7918e647: Log is configured to *not* fsync() on all Append() calls
I20260926 06:15:19.608047 7988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e78099f3e87a46d886bb111a7918e647: No bootstrap required, opened a new log
I20260926 06:15:19.609668 7988 raft_consensus.cc:348] T 00000000000000000000000000000000 P e78099f3e87a46d886bb111a7918e647 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e78099f3e87a46d886bb111a7918e647" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 40043 } }
I20260926 06:15:19.609923 7988 raft_consensus.cc:374] T 00000000000000000000000000000000 P e78099f3e87a46d886bb111a7918e647 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:19.609993 7988 raft_consensus.cc:729] T 00000000000000000000000000000000 P e78099f3e87a46d886bb111a7918e647 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e78099f3e87a46d886bb111a7918e647, State: Initialized, Role: FOLLOWER
I20260926 06:15:19.610146 7988 consensus_queue.cc:260] T 00000000000000000000000000000000 P e78099f3e87a46d886bb111a7918e647 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e78099f3e87a46d886bb111a7918e647" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 40043 } }
I20260926 06:15:19.610244 7988 raft_consensus.cc:388] T 00000000000000000000000000000000 P e78099f3e87a46d886bb111a7918e647 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:15:19.610286 7988 raft_consensus.cc:482] T 00000000000000000000000000000000 P e78099f3e87a46d886bb111a7918e647 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:15:19.610388 7988 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e78099f3e87a46d886bb111a7918e647 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:19.611833 7988 raft_consensus.cc:504] T 00000000000000000000000000000000 P e78099f3e87a46d886bb111a7918e647 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e78099f3e87a46d886bb111a7918e647" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 40043 } }
I20260926 06:15:19.612003 7988 leader_election.cc:302] T 00000000000000000000000000000000 P e78099f3e87a46d886bb111a7918e647 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e78099f3e87a46d886bb111a7918e647; no voters:
I20260926 06:15:19.612066 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 7921
I20260926 06:15:19.612164 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790403292369990-2491-0/minicluster-data/master-0/wal/instance
I20260926 06:15:19.612218 7988 leader_election.cc:288] T 00000000000000000000000000000000 P e78099f3e87a46d886bb111a7918e647 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:15:19.612300 7991 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e78099f3e87a46d886bb111a7918e647 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:15:19.612372 2491 external_mini_cluster.cc:1428] Setting key 324f0cd8849de4e70dae16ccc0dd0c35
I20260926 06:15:19.612526 7988 sys_catalog.cc:564] T 00000000000000000000000000000000 P e78099f3e87a46d886bb111a7918e647 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:15:19.612564 7991 raft_consensus.cc:686] T 00000000000000000000000000000000 P e78099f3e87a46d886bb111a7918e647 [term 1 LEADER]: Becoming Leader. State: Replica: e78099f3e87a46d886bb111a7918e647, State: Running, Role: LEADER
I20260926 06:15:19.612746 7991 consensus_queue.cc:237] T 00000000000000000000000000000000 P e78099f3e87a46d886bb111a7918e647 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e78099f3e87a46d886bb111a7918e647" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 40043 } }
I20260926 06:15:19.613327 7991 sys_catalog.cc:455] T 00000000000000000000000000000000 P e78099f3e87a46d886bb111a7918e647 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e78099f3e87a46d886bb111a7918e647" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e78099f3e87a46d886bb111a7918e647" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 40043 } } }
I20260926 06:15:19.613456 7991 sys_catalog.cc:458] T 00000000000000000000000000000000 P e78099f3e87a46d886bb111a7918e647 [sys.catalog]: This master's current role is: LEADER
I20260926 06:15:19.613610 7991 sys_catalog.cc:455] T 00000000000000000000000000000000 P e78099f3e87a46d886bb111a7918e647 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e78099f3e87a46d886bb111a7918e647. Latest consensus state: current_term: 1 leader_uuid: "e78099f3e87a46d886bb111a7918e647" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e78099f3e87a46d886bb111a7918e647" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 40043 } } }
I20260926 06:15:19.613665 7991 sys_catalog.cc:458] T 00000000000000000000000000000000 P e78099f3e87a46d886bb111a7918e647 [sys.catalog]: This master's current role is: LEADER
I20260926 06:15:19.613857 8002 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:15:19.614400 8002 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:15:19.616003 8002 catalog_manager.cc:1347] Generated new cluster ID: 3303c249615c40c6940852a4ff8f5e4d
I20260926 06:15:19.616060 8002 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:15:19.620333 8001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:19.613195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38950 (local address 127.2.110.254:40043)
0926 06:15:19.613533 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:19.613544 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:19.613816 (+ 272us) server_negotiation.cc:407] Connection header received
0926 06:15:19.613963 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:19.613971 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:19.614055 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:19.614218 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:19.614579 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:19.616063 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:19.617177 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:19.617840 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:19.617977 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:19.619306 (+ 1329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:19.619310 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:19.619311 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:15:19.619539 (+ 228us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:19.619584 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:15:19.619601 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:19.619670 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:19.619993 (+ 323us) server_negotiation.cc:299] Negotiation successful
0926 06:15:19.620092 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":64,"threads_started":1}
I20260926 06:15:19.621193 2491 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:15:19.624047 8002 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:15:19.624606 8002 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:15:19.629416 8002 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e78099f3e87a46d886bb111a7918e647: Generated new TSK 0
I20260926 06:15:19.629720 8002 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:15:19.674077 8001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:19.638690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38956 (local address 127.2.110.254:40043)
0926 06:15:19.638745 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:19.638747 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:19.638851 (+ 104us) server_negotiation.cc:407] Connection header received
0926 06:15:19.638942 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:19.638943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:19.638958 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:19.639040 (+ 82us) server_negotiation.cc:226] Negotiated authn=JWT
0926 06:15:19.639218 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:19.640029 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:19.640869 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:19.641195 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:19.641272 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:19.641876 (+ 604us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 06:15:19.673771 (+ 31895us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 06:15:19.673991 (+ 220us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38956: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":27}
W20260926 06:15:19.674165 8001 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38956: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 06:15:19.674579 8017 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:19.638581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:40043 (local address 127.0.0.1:38956)
0926 06:15:19.638777 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:19.638788 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:19.638848 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:19.639082 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:19.639082 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:19.639084 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 06:15:19.639139 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:19.639140 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:19.640161 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:19.640162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:19.640775 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:19.640777 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:19.641578 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:19.641581 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:19.641675 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:19.641686 (+ 11us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 06:15:19.674077 (+ 32391us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 06:15:19.674500 (+ 423us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.110.254:40043: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":114,"threads_started":1}
I20260926 06:15:19.675016 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 7921
2026-09-26T06:15:19Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (234 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (98 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (94 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:15:19 dist-test-slave-hndd krb5kdc[8029](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:15:19 dist-test-slave-hndd krb5kdc[8029](info): set up 2 sockets
Sep 26 06:15:19 dist-test-slave-hndd krb5kdc[8029](info): commencing operation
krb5kdc: starting...
W20260926 06:15:20.953923 2491 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Sep 26 06:15:21 dist-test-slave-hndd krb5kdc[8029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:15:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:15:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
I20260926 06:15:21.086150 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:34851
--webserver_interface=127.2.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:40903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.110.254:34851
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:15:21.173676 8042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:21.173919 8042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:21.173945 8042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:21.175225 8042 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:15:21.175276 8042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:21.175292 8042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:21.175304 8042 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:15:21.175318 8042 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:15:21.176862 8042 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:40903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.110.254:34851
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:34851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:21.177156 8042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:21.177428 8042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:21.182811 8050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:21.182811 8047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:21.182868 8048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:21.183043 8042 server_base.cc:1034] running on GCE node
I20260926 06:15:21.183420 8042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:21.183730 8042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:21.184918 8042 hybrid_clock.cc:648] HybridClock initialized: now 1790403321184897 us; error 40 us; skew 500 ppm
Sep 26 06:15:21 dist-test-slave-hndd krb5kdc[8029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403321, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:21.192682 8042 init.cc:377] Logged in from keytab as kudu/127.2.110.254@KRBTEST.COM (short username kudu)
I20260926 06:15:21.193418 8042 webserver.cc:460] Webserver started at http://127.2.110.254:35027/ using document root <none> and password file <none>
I20260926 06:15:21.193677 8042 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:21.193748 8042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:21.193899 8042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:21.195051 8042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/master-0/data/instance:
uuid: "558dfb38136a47eabdc13e7bf9fea312"
format_stamp: "Formatted at 2026-09-26 06:15:21 on dist-test-slave-hndd"
server_key: "12cb7e2a9f4981fd6855e316ef4953a0"
server_key_iv: "c969587008bbce63f832f7c43a1ea8c6"
server_key_version: "encryptionkey@0"
I20260926 06:15:21.195444 8042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/master-0/wal/instance:
uuid: "558dfb38136a47eabdc13e7bf9fea312"
format_stamp: "Formatted at 2026-09-26 06:15:21 on dist-test-slave-hndd"
server_key: "12cb7e2a9f4981fd6855e316ef4953a0"
server_key_iv: "c969587008bbce63f832f7c43a1ea8c6"
server_key_version: "encryptionkey@0"
I20260926 06:15:21.196965 8042 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:15:21.198215 8057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:21.198475 8042 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 06:15:21.198598 8042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "558dfb38136a47eabdc13e7bf9fea312"
format_stamp: "Formatted at 2026-09-26 06:15:21 on dist-test-slave-hndd"
server_key: "12cb7e2a9f4981fd6855e316ef4953a0"
server_key_iv: "c969587008bbce63f832f7c43a1ea8c6"
server_key_version: "encryptionkey@0"
I20260926 06:15:21.198704 8042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:21.213963 8042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:21.216399 8042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:21.216598 8042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:21.221053 8042 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.254:34851
I20260926 06:15:21.221447 8109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.254:34851 every 8 connection(s)
I20260926 06:15:21.221544 8042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:15:21.222499 8110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:21.222821 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 8042
I20260926 06:15:21.222935 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/master-0/wal/instance
I20260926 06:15:21.223133 2491 external_mini_cluster.cc:1428] Setting key 38e15400b563abd7427fc93cc563798a
I20260926 06:15:21.225776 8110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 558dfb38136a47eabdc13e7bf9fea312: Bootstrap starting.
I20260926 06:15:21.226423 8110 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 558dfb38136a47eabdc13e7bf9fea312: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:21.226759 8110 log.cc:824] T 00000000000000000000000000000000 P 558dfb38136a47eabdc13e7bf9fea312: Log is configured to *not* fsync() on all Append() calls
I20260926 06:15:21.227487 8110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 558dfb38136a47eabdc13e7bf9fea312: No bootstrap required, opened a new log
I20260926 06:15:21.230096 8110 raft_consensus.cc:348] T 00000000000000000000000000000000 P 558dfb38136a47eabdc13e7bf9fea312 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "558dfb38136a47eabdc13e7bf9fea312" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 34851 } }
I20260926 06:15:21.230293 8110 raft_consensus.cc:374] T 00000000000000000000000000000000 P 558dfb38136a47eabdc13e7bf9fea312 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:21.230330 8110 raft_consensus.cc:729] T 00000000000000000000000000000000 P 558dfb38136a47eabdc13e7bf9fea312 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 558dfb38136a47eabdc13e7bf9fea312, State: Initialized, Role: FOLLOWER
I20260926 06:15:21.230477 8110 consensus_queue.cc:260] T 00000000000000000000000000000000 P 558dfb38136a47eabdc13e7bf9fea312 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "558dfb38136a47eabdc13e7bf9fea312" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 34851 } }
I20260926 06:15:21.230564 8110 raft_consensus.cc:388] T 00000000000000000000000000000000 P 558dfb38136a47eabdc13e7bf9fea312 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:15:21.230597 8110 raft_consensus.cc:482] T 00000000000000000000000000000000 P 558dfb38136a47eabdc13e7bf9fea312 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:15:21.230654 8110 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 558dfb38136a47eabdc13e7bf9fea312 [term 0 FOLLOWER]: Advancing to term 1
Sep 26 06:15:21 dist-test-slave-hndd krb5kdc[8029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:15:21.231484 8110 raft_consensus.cc:504] T 00000000000000000000000000000000 P 558dfb38136a47eabdc13e7bf9fea312 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "558dfb38136a47eabdc13e7bf9fea312" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 34851 } }
I20260926 06:15:21.231594 8110 leader_election.cc:302] T 00000000000000000000000000000000 P 558dfb38136a47eabdc13e7bf9fea312 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 558dfb38136a47eabdc13e7bf9fea312; no voters:
I20260926 06:15:21.231859 8110 leader_election.cc:288] T 00000000000000000000000000000000 P 558dfb38136a47eabdc13e7bf9fea312 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:15:21.231925 8115 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 558dfb38136a47eabdc13e7bf9fea312 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:15:21.232122 8115 raft_consensus.cc:686] T 00000000000000000000000000000000 P 558dfb38136a47eabdc13e7bf9fea312 [term 1 LEADER]: Becoming Leader. State: Replica: 558dfb38136a47eabdc13e7bf9fea312, State: Running, Role: LEADER
I20260926 06:15:21.232167 8110 sys_catalog.cc:564] T 00000000000000000000000000000000 P 558dfb38136a47eabdc13e7bf9fea312 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:15:21.232234 8115 consensus_queue.cc:237] T 00000000000000000000000000000000 P 558dfb38136a47eabdc13e7bf9fea312 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "558dfb38136a47eabdc13e7bf9fea312" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 34851 } }
I20260926 06:15:21.232728 8117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 558dfb38136a47eabdc13e7bf9fea312 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "558dfb38136a47eabdc13e7bf9fea312" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "558dfb38136a47eabdc13e7bf9fea312" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 34851 } } }
I20260926 06:15:21.232867 8117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 558dfb38136a47eabdc13e7bf9fea312 [sys.catalog]: This master's current role is: LEADER
I20260926 06:15:21.232892 8116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 558dfb38136a47eabdc13e7bf9fea312 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 558dfb38136a47eabdc13e7bf9fea312. Latest consensus state: current_term: 1 leader_uuid: "558dfb38136a47eabdc13e7bf9fea312" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "558dfb38136a47eabdc13e7bf9fea312" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 34851 } } }
I20260926 06:15:21.232990 8116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 558dfb38136a47eabdc13e7bf9fea312 [sys.catalog]: This master's current role is: LEADER
I20260926 06:15:21.233222 8124 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:15:21.233865 8124 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:15:21.235688 8113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:21.223759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45798 (local address 127.2.110.254:34851)
0926 06:15:21.223953 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:21.223962 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:21.223997 (+ 35us) server_negotiation.cc:407] Connection header received
0926 06:15:21.224414 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:21.224420 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:21.224473 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:21.224591 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:21.225544 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:21.227038 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:21.228015 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:21.228499 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:21.228641 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:21.231517 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:21.231521 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:21.231521 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:21.231533 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:21.233991 (+ 2458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:21.234408 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:21.234411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:21.234412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:21.234518 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:21.234759 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:21.234760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:21.234761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:21.235042 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:21.235235 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:21.235345 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 06:15:21.235503 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":104,"threads_started":1}
I20260926 06:15:21.237936 8124 catalog_manager.cc:1347] Generated new cluster ID: ba74118f70bb47429230b0910f7c128b
I20260926 06:15:21.238044 8124 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:15:21.248133 8124 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:15:21.248847 8124 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:15:21.254174 8124 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 558dfb38136a47eabdc13e7bf9fea312: Generated new TSK 0
I20260926 06:15:21.254431 8124 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab.
Entry for principal kudu/127.2.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab.
I20260926 06:15:21.293198 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.193:0
--local_ip_for_outbound_sockets=127.2.110.193
--webserver_interface=127.2.110.193
--webserver_port=0
--tserver_master_addrs=127.2.110.254:34851
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab
--principal=kudu/127.2.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:40903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:15:21.386650 8136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:21.386896 8136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:21.386924 8136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:21.388175 8136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:21.388224 8136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:21.388254 8136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.193
I20260926 06:15:21.389959 8136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:40903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab
--principal=kudu/127.2.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.110.193
--webserver_port=0
--tserver_master_addrs=127.2.110.254:34851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.193
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:21.390197 8136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:21.390406 8136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:21.395601 8141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:21.395603 8144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:21.395601 8142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:21.395896 8136 server_base.cc:1034] running on GCE node
I20260926 06:15:21.396159 8136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:21.396482 8136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:21.397693 8136 hybrid_clock.cc:648] HybridClock initialized: now 1790403321397657 us; error 38 us; skew 500 ppm
Sep 26 06:15:21 dist-test-slave-hndd krb5kdc[8029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403321, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:21.400452 8136 init.cc:377] Logged in from keytab as kudu/127.2.110.193@KRBTEST.COM (short username kudu)
I20260926 06:15:21.401090 8136 webserver.cc:460] Webserver started at http://127.2.110.193:38981/ using document root <none> and password file <none>
I20260926 06:15:21.401432 8136 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:21.401487 8136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:21.401641 8136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:21.402882 8136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/ts-0/data/instance:
uuid: "78dc18aaf9bf4724adba4fcd865e04f8"
format_stamp: "Formatted at 2026-09-26 06:15:21 on dist-test-slave-hndd"
server_key: "ccd0f3d4b7f338ca36508843558a5f22"
server_key_iv: "39a6f71373ec6cda4d502d3356fb46a6"
server_key_version: "encryptionkey@0"
I20260926 06:15:21.403321 8136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/ts-0/wal/instance:
uuid: "78dc18aaf9bf4724adba4fcd865e04f8"
format_stamp: "Formatted at 2026-09-26 06:15:21 on dist-test-slave-hndd"
server_key: "ccd0f3d4b7f338ca36508843558a5f22"
server_key_iv: "39a6f71373ec6cda4d502d3356fb46a6"
server_key_version: "encryptionkey@0"
I20260926 06:15:21.404808 8136 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:15:21.405922 8151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:21.406176 8136 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:15:21.406265 8136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/ts-0/wal
uuid: "78dc18aaf9bf4724adba4fcd865e04f8"
format_stamp: "Formatted at 2026-09-26 06:15:21 on dist-test-slave-hndd"
server_key: "ccd0f3d4b7f338ca36508843558a5f22"
server_key_iv: "39a6f71373ec6cda4d502d3356fb46a6"
server_key_version: "encryptionkey@0"
I20260926 06:15:21.406369 8136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:21.417304 8136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:21.419874 8136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:21.420001 8136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:21.420307 8136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:15:21.420642 8136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:15:21.420699 8136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:21.420743 8136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:15:21.420771 8136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:21.428344 8136 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.193:41991
I20260926 06:15:21.428439 8264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.193:41991 every 8 connection(s)
I20260926 06:15:21.428814 8136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
I20260926 06:15:21.429997 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 8136
I20260926 06:15:21.430137 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/ts-0/wal/instance
I20260926 06:15:21.430346 2491 external_mini_cluster.cc:1428] Setting key e6fad9fe9dd912e01c7aa2697fa07508
Sep 26 06:15:21 dist-test-slave-hndd krb5kdc[8029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403321, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.193@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.110.194@KRBTEST.COM; defaulting to no policy
I20260926 06:15:21.440042 8113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:21.429704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:39249 (local address 127.2.110.254:34851)
0926 06:15:21.429805 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:21.429808 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:21.430015 (+ 207us) server_negotiation.cc:407] Connection header received
0926 06:15:21.431222 (+ 1207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:21.431223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:21.431236 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:21.431331 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:21.432461 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:21.433144 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:21.434314 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:21.434530 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:21.434613 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:21.436835 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:21.436837 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:21.436837 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:21.436846 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:21.438432 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:21.438954 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:21.438955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:21.438956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:21.439026 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:21.439303 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:21.439304 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:21.439304 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:21.439440 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:21.439519 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:21.439843 (+ 324us) server_negotiation.cc:299] Negotiation successful
0926 06:15:21.439936 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 06:15:21.440263 8267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:21.429687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34851 (local address 127.2.110.193:39249)
0926 06:15:21.429867 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:21.429912 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:21.431068 (+ 1156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:21.431397 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:21.431401 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:21.431677 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:21.432339 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:21.432351 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:21.433298 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:21.433299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:21.434200 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:21.434202 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:21.434624 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:21.434625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:21.434768 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:21.435341 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:21.435349 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:21.436732 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:21.438544 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:21.438552 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:21.438555 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:21.438858 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:21.439097 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:21.439097 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:21.439098 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:21.439226 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:21.439510 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:21.439517 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:21.439702 (+ 185us) client_negotiation.cc:769] Sending connection context
0926 06:15:21.439824 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 06:15:21.440022 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":101,"threads_started":1}
I20260926 06:15:21.440894 8265 heartbeater.cc:347] Connected to a master server at 127.2.110.254:34851
I20260926 06:15:21.441013 8265 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.194@KRBTEST.COM" created.
I20260926 06:15:21.441233 8265 heartbeater.cc:511] Master 127.2.110.254:34851 requested a full tablet report, sending...
I20260926 06:15:21.441932 8074 ts_manager.cc:194] Registered new tserver with Master: 78dc18aaf9bf4724adba4fcd865e04f8 (127.2.110.193:41991)
I20260926 06:15:21.442857 8074 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.193@KRBTEST.COM'} at 127.2.110.193:39249
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab.
Entry for principal kudu/127.2.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab.
I20260926 06:15:21.462164 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.194:0
--local_ip_for_outbound_sockets=127.2.110.194
--webserver_interface=127.2.110.194
--webserver_port=0
--tserver_master_addrs=127.2.110.254:34851
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab
--principal=kudu/127.2.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:40903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:15:21.547887 8270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:21.548139 8270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:21.548178 8270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:21.549369 8270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:21.549448 8270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:21.549495 8270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.194
I20260926 06:15:21.551064 8270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:40903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab
--principal=kudu/127.2.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.110.194
--webserver_port=0
--tserver_master_addrs=127.2.110.254:34851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.194
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:21.551337 8270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:21.551594 8270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:21.556958 8275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:21.557103 8270 server_base.cc:1034] running on GCE node
W20260926 06:15:21.557163 8278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:21.556991 8276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:21.557461 8270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:21.557672 8270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:21.558854 8270 hybrid_clock.cc:648] HybridClock initialized: now 1790403321558854 us; error 51 us; skew 500 ppm
Sep 26 06:15:21 dist-test-slave-hndd krb5kdc[8029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403321, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:21.561715 8270 init.cc:377] Logged in from keytab as kudu/127.2.110.194@KRBTEST.COM (short username kudu)
I20260926 06:15:21.562528 8270 webserver.cc:460] Webserver started at http://127.2.110.194:37761/ using document root <none> and password file <none>
I20260926 06:15:21.562798 8270 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:21.562871 8270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:21.563030 8270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:21.564111 8270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/ts-1/data/instance:
uuid: "f47cca766e9545b990317d035904be75"
format_stamp: "Formatted at 2026-09-26 06:15:21 on dist-test-slave-hndd"
server_key: "8b9cbf999014051d363fad68e1549992"
server_key_iv: "57cff6a9de5fd2a69e63a634b45d9688"
server_key_version: "encryptionkey@0"
I20260926 06:15:21.564479 8270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/ts-1/wal/instance:
uuid: "f47cca766e9545b990317d035904be75"
format_stamp: "Formatted at 2026-09-26 06:15:21 on dist-test-slave-hndd"
server_key: "8b9cbf999014051d363fad68e1549992"
server_key_iv: "57cff6a9de5fd2a69e63a634b45d9688"
server_key_version: "encryptionkey@0"
I20260926 06:15:21.565974 8270 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:15:21.566898 8285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:21.567080 8270 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:15:21.567171 8270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/ts-1/wal
uuid: "f47cca766e9545b990317d035904be75"
format_stamp: "Formatted at 2026-09-26 06:15:21 on dist-test-slave-hndd"
server_key: "8b9cbf999014051d363fad68e1549992"
server_key_iv: "57cff6a9de5fd2a69e63a634b45d9688"
server_key_version: "encryptionkey@0"
I20260926 06:15:21.567229 8270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:21.617060 8270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:21.620402 8270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:21.620577 8270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:21.620901 8270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:15:21.621330 8270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:15:21.621428 8270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:21.621476 8270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:15:21.621507 8270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:21.628432 8270 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.194:40225
I20260926 06:15:21.628556 8398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.194:40225 every 8 connection(s)
I20260926 06:15:21.628851 8270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
Sep 26 06:15:21 dist-test-slave-hndd krb5kdc[8029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403321, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.194@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:15:21.638289 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 8270
I20260926 06:15:21.638401 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/ts-1/wal/instance
I20260926 06:15:21.638585 2491 external_mini_cluster.cc:1428] Setting key a1b695b3ba3e2f371c158742cb7eb3b8
I20260926 06:15:21.640031 8113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:21.629705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:52445 (local address 127.2.110.254:34851)
0926 06:15:21.629797 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:21.629800 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:21.629984 (+ 184us) server_negotiation.cc:407] Connection header received
0926 06:15:21.630915 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:21.630917 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:21.630927 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:21.630983 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:21.631655 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:21.632435 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:21.633955 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:21.634177 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:21.634257 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:21.636589 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:21.636590 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:21.636590 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:21.636597 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:21.638210 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:21.638893 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:21.638895 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:21.638895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:21.638982 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:21.639282 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:21.639283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:21.639284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:21.639403 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:21.639493 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:21.639844 (+ 351us) server_negotiation.cc:299] Negotiation successful
0926 06:15:21.639931 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 06:15:21.640262 8401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:21.629665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34851 (local address 127.2.110.194:52445)
0926 06:15:21.629856 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:21.629892 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:21.630803 (+ 911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:21.631035 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:21.631039 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:21.631301 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:21.631560 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:21.631565 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:21.632711 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:21.632713 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:21.633850 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:21.633854 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:21.634323 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:21.634325 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:21.634479 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:21.635028 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:21.635036 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:21.636490 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:21.638320 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:21.638326 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:21.638328 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:21.638785 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:21.639068 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:21.639069 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:21.639069 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:21.639191 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:21.639498 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:21.639504 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:21.639685 (+ 181us) client_negotiation.cc:769] Sending connection context
0926 06:15:21.639809 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 06:15:21.639994 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":105,"threads_started":1}
I20260926 06:15:21.640795 8399 heartbeater.cc:347] Connected to a master server at 127.2.110.254:34851
I20260926 06:15:21.640892 8399 heartbeater.cc:464] Registering TS with master...
I20260926 06:15:21.641091 8399 heartbeater.cc:511] Master 127.2.110.254:34851 requested a full tablet report, sending...
I20260926 06:15:21.641669 8074 ts_manager.cc:194] Registered new tserver with Master: f47cca766e9545b990317d035904be75 (127.2.110.194:40225)
I20260926 06:15:21.642218 8074 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.194@KRBTEST.COM'} at 127.2.110.194:52445
WARNING: no policy specified for kudu/127.2.110.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab.
Entry for principal kudu/127.2.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab.
I20260926 06:15:21.664680 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.195:0
--local_ip_for_outbound_sockets=127.2.110.195
--webserver_interface=127.2.110.195
--webserver_port=0
--tserver_master_addrs=127.2.110.254:34851
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab
--principal=kudu/127.2.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:40903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:15:21.750823 8404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:21.751127 8404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:21.751183 8404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:21.752815 8404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:21.752957 8404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:21.753028 8404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.195
I20260926 06:15:21.756716 8404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:40903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab
--principal=kudu/127.2.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.110.195
--webserver_port=0
--tserver_master_addrs=127.2.110.254:34851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.195
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:21.757107 8404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:21.757457 8404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:21.762595 8409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:21.762601 8410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:21.762598 8412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:21.763037 8404 server_base.cc:1034] running on GCE node
I20260926 06:15:21.763453 8404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:21.763784 8404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:21.764982 8404 hybrid_clock.cc:648] HybridClock initialized: now 1790403321764966 us; error 31 us; skew 500 ppm
Sep 26 06:15:21 dist-test-slave-hndd krb5kdc[8029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403321, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:21.767846 8404 init.cc:377] Logged in from keytab as kudu/127.2.110.195@KRBTEST.COM (short username kudu)
I20260926 06:15:21.768429 8404 webserver.cc:460] Webserver started at http://127.2.110.195:40353/ using document root <none> and password file <none>
I20260926 06:15:21.768707 8404 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:21.768787 8404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:21.768955 8404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:21.770228 8404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/ts-2/data/instance:
uuid: "ad89cb59d2ee4c268a144885db2daca1"
format_stamp: "Formatted at 2026-09-26 06:15:21 on dist-test-slave-hndd"
server_key: "dd8f079a6713b85fb052925b9b37df0d"
server_key_iv: "22951befee963697728a57ea267b9f5c"
server_key_version: "encryptionkey@0"
I20260926 06:15:21.770617 8404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/ts-2/wal/instance:
uuid: "ad89cb59d2ee4c268a144885db2daca1"
format_stamp: "Formatted at 2026-09-26 06:15:21 on dist-test-slave-hndd"
server_key: "dd8f079a6713b85fb052925b9b37df0d"
server_key_iv: "22951befee963697728a57ea267b9f5c"
server_key_version: "encryptionkey@0"
I20260926 06:15:21.772229 8404 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 06:15:21.773247 8419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:21.773466 8404 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:15:21.773576 8404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/ts-2/wal
uuid: "ad89cb59d2ee4c268a144885db2daca1"
format_stamp: "Formatted at 2026-09-26 06:15:21 on dist-test-slave-hndd"
server_key: "dd8f079a6713b85fb052925b9b37df0d"
server_key_iv: "22951befee963697728a57ea267b9f5c"
server_key_version: "encryptionkey@0"
I20260926 06:15:21.773672 8404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:21.782234 8404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:21.784487 8404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:21.784688 8404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:21.785068 8404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:15:21.785480 8404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:15:21.785535 8404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:21.785579 8404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:15:21.785605 8404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:21.792344 8404 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.195:38287
I20260926 06:15:21.792405 8532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.195:38287 every 8 connection(s)
I20260926 06:15:21.792742 8404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
Sep 26 06:15:21 dist-test-slave-hndd krb5kdc[8029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403321, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.195@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:15:21.800107 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 8404
I20260926 06:15:21.800227 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/minicluster-data/ts-2/wal/instance
I20260926 06:15:21.800410 2491 external_mini_cluster.cc:1428] Setting key f7a52db04d3992759a78b871b11df527
I20260926 06:15:21.803606 8113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:21.793587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:36411 (local address 127.2.110.254:34851)
0926 06:15:21.793640 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:21.793642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:21.793832 (+ 190us) server_negotiation.cc:407] Connection header received
0926 06:15:21.794887 (+ 1055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:21.794889 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:21.794905 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:21.795005 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:21.795693 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:21.796255 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:21.797291 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:21.797702 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:21.797793 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:21.800484 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:21.800485 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:21.800486 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:21.800494 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:21.801992 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:21.802430 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:21.802431 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:21.802431 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:21.802488 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:21.802830 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:21.802833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:21.802833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:21.803038 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:21.803151 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:21.803442 (+ 291us) server_negotiation.cc:299] Negotiation successful
0926 06:15:21.803529 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 06:15:21.803853 8535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:21.793549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34851 (local address 127.2.110.195:36411)
0926 06:15:21.793703 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:21.793746 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:21.794701 (+ 955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:21.795081 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:21.795084 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:21.795347 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:21.795549 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:21.795553 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:21.796356 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:21.796357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:21.797180 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:21.797182 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:21.797888 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:21.797890 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:21.798086 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:21.798816 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:21.798826 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:21.800377 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:21.802089 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:21.802096 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:21.802098 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:21.802363 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:21.802547 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:21.802547 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:21.802547 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:21.802648 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:21.803135 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:21.803140 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:21.803283 (+ 143us) client_negotiation.cc:769] Sending connection context
0926 06:15:21.803378 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 06:15:21.803520 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":87,"threads_started":1}
I20260926 06:15:21.804450 8533 heartbeater.cc:347] Connected to a master server at 127.2.110.254:34851
I20260926 06:15:21.804576 8533 heartbeater.cc:464] Registering TS with master...
I20260926 06:15:21.804777 8533 heartbeater.cc:511] Master 127.2.110.254:34851 requested a full tablet report, sending...
I20260926 06:15:21.805194 8074 ts_manager.cc:194] Registered new tserver with Master: ad89cb59d2ee4c268a144885db2daca1 (127.2.110.195:38287)
I20260926 06:15:21.805626 8074 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.195@KRBTEST.COM'} at 127.2.110.195:36411
I20260926 06:15:21.812866 2491 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/krb5kdc/test-admin.keytab.
Sep 26 06:15:21 dist-test-slave-hndd krb5kdc[8029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:21.830886 2491 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 06:15:21.832927 8540 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:21.831940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35196 (local address 127.2.110.193:41991)
0926 06:15:21.832262 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:21.832267 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:21.832297 (+ 30us) server_negotiation.cc:407] Connection header received
0926 06:15:21.832326 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:21.832328 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:21.832344 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:21.832408 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:21.832824 (+ 416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35196: BlockingRecv error: recv got EOF from 127.0.0.1:35196 (error 108)
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":225,"threads_started":1}
W20260926 06:15:21.833014 2491 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/krb5kdc/krb5cc' of type FILE
Sep 26 06:15:21 dist-test-slave-hndd krb5kdc[8029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:21.834231 2491 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 06:15:21 dist-test-slave-hndd krb5kdc[8029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:21.835297 2491 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 06:15:21 dist-test-slave-hndd krb5kdc[8029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.110.193@KRBTEST.COM
I20260926 06:15:21.844938 8540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:21.835941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35200 (local address 127.2.110.193:41991)
0926 06:15:21.836021 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:21.836023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:21.836034 (+ 11us) server_negotiation.cc:407] Connection header received
0926 06:15:21.836082 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:21.836083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:21.836095 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:21.836156 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:21.836573 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:21.837406 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:21.838252 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:21.838623 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:21.838692 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:21.841495 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:21.841505 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:21.841506 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:21.841514 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:21.843383 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:21.843826 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:21.843829 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:21.843829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:21.843904 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:21.844185 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:21.844186 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:21.844186 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:21.844336 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:21.844469 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:21.844563 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 06:15:21.844644 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260926 06:15:21.847805 8540 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:21.846873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35208 (local address 127.2.110.193:41991)
0926 06:15:21.846977 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:21.846979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:21.847015 (+ 36us) server_negotiation.cc:407] Connection header received
0926 06:15:21.847113 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:21.847114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:21.847125 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:21.847194 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:21.847654 (+ 460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35208: BlockingRecv error: recv got EOF from 127.0.0.1:35208 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260926 06:15:21.848007 2491 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/krb5kdc/krb5cc' of type FILE
Sep 26 06:15:21 dist-test-slave-hndd krb5kdc[8029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:21.849516 2491 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 06:15:21 dist-test-slave-hndd krb5kdc[8029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:21.850621 2491 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 06:15:21 dist-test-slave-hndd krb5kdc[8029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.110.193@KRBTEST.COM
I20260926 06:15:21.858719 8540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:21.851238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35214 (local address 127.2.110.193:41991)
0926 06:15:21.851313 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:21.851314 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:21.851341 (+ 27us) server_negotiation.cc:407] Connection header received
0926 06:15:21.851361 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:21.851362 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:21.851371 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:21.851411 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:21.851843 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:21.852542 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:21.853416 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:21.853634 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:21.853702 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:21.856042 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:21.856043 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:21.856044 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:21.856051 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:21.857651 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:21.858023 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:21.858024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:21.858024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:21.858076 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:21.858293 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:21.858294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:21.858294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:21.858397 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:21.858473 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:21.858580 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 06:15:21.858645 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260926 06:15:21.860432 8540 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:21.859874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35224 (local address 127.2.110.193:41991)
0926 06:15:21.859969 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:21.859971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:21.859999 (+ 28us) server_negotiation.cc:407] Connection header received
0926 06:15:21.860052 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:21.860052 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:21.860061 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:21.860113 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:21.860389 (+ 276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35224: BlockingRecv error: recv got EOF from 127.0.0.1:35224 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260926 06:15:21.860520 2491 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403292369990-2491-0/krb5kdc/krb5cc' of type FILE
Sep 26 06:15:21 dist-test-slave-hndd krb5kdc[8029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:21.861642 2491 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 06:15:21 dist-test-slave-hndd krb5kdc[8029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:21.862572 2491 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 06:15:21 dist-test-slave-hndd krb5kdc[8029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.110.193@KRBTEST.COM
I20260926 06:15:21.871824 8540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:21.863198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35238 (local address 127.2.110.193:41991)
0926 06:15:21.863265 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:21.863266 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:21.863293 (+ 27us) server_negotiation.cc:407] Connection header received
0926 06:15:21.863346 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:21.863346 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:21.863355 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:21.863411 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:21.863843 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:21.864331 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:21.865092 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:21.865323 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:21.865542 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:21.868075 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:21.868077 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:21.868078 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:21.868088 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:21.869499 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:21.870223 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:21.870224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:21.870225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:21.870394 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:21.870893 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:21.870895 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:21.870895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:21.871152 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:21.871326 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:21.871503 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 06:15:21.871609 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 06:15:21.872694 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 8136
I20260926 06:15:21.879083 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 8270
I20260926 06:15:21.883992 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 8404
I20260926 06:15:21.889042 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 8042
2026-09-26T06:15:21Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2023 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:15:21 dist-test-slave-hndd krb5kdc[8553](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:15:21 dist-test-slave-hndd krb5kdc[8553](info): set up 2 sockets
Sep 26 06:15:21 dist-test-slave-hndd krb5kdc[8553](info): commencing operation
krb5kdc: starting...
W20260926 06:15:22.941870 2491 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Sep 26 06:15:23 dist-test-slave-hndd krb5kdc[8553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:15:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:15:23Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/oryx_127.2.110.254.keytab.
Entry for principal oryx/127.2.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/oryx_127.2.110.254.keytab.
I20260926 06:15:23.074831 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:46749
--webserver_interface=127.2.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/oryx_127.2.110.254.keytab
--principal=oryx/127.2.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:41555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.110.254:46749
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:15:23.158073 8566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:23.158313 8566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:23.158339 8566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:23.159570 8566 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:15:23.159622 8566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:23.159637 8566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:23.159651 8566 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:15:23.159663 8566 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:15:23.161387 8566 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:41555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.110.254:46749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/oryx_127.2.110.254.keytab
--principal=oryx/127.2.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:46749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:23.161695 8566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:23.161980 8566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:23.167419 8571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:23.167606 8566 server_base.cc:1034] running on GCE node
W20260926 06:15:23.167438 8574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:23.167446 8572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:23.167999 8566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:23.168368 8566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:23.169528 8566 hybrid_clock.cc:648] HybridClock initialized: now 1790403323169507 us; error 47 us; skew 500 ppm
Sep 26 06:15:23 dist-test-slave-hndd krb5kdc[8553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403323, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:23.172432 8566 init.cc:377] Logged in from keytab as oryx/127.2.110.254@KRBTEST.COM (short username oryx)
I20260926 06:15:23.173012 8566 webserver.cc:460] Webserver started at http://127.2.110.254:36329/ using document root <none> and password file <none>
I20260926 06:15:23.173309 8566 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:23.173440 8566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:23.173636 8566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:23.175494 8566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-0/data/instance:
uuid: "eed8ee75c5394469994882db1a963644"
format_stamp: "Formatted at 2026-09-26 06:15:23 on dist-test-slave-hndd"
server_key: "f702a73419f0a237e9b74490606f3d6c"
server_key_iv: "026727b2ea43a47b1211390ab1923151"
server_key_version: "encryptionkey@0"
I20260926 06:15:23.175894 8566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-0/wal/instance:
uuid: "eed8ee75c5394469994882db1a963644"
format_stamp: "Formatted at 2026-09-26 06:15:23 on dist-test-slave-hndd"
server_key: "f702a73419f0a237e9b74490606f3d6c"
server_key_iv: "026727b2ea43a47b1211390ab1923151"
server_key_version: "encryptionkey@0"
I20260926 06:15:23.177654 8566 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260926 06:15:23.178544 8581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:23.178900 8566 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:23.178998 8566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "eed8ee75c5394469994882db1a963644"
format_stamp: "Formatted at 2026-09-26 06:15:23 on dist-test-slave-hndd"
server_key: "f702a73419f0a237e9b74490606f3d6c"
server_key_iv: "026727b2ea43a47b1211390ab1923151"
server_key_version: "encryptionkey@0"
I20260926 06:15:23.179057 8566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:23.188587 8566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:23.190826 8566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:23.190999 8566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:23.195089 8566 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.254:46749
I20260926 06:15:23.195160 8633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.254:46749 every 8 connection(s)
I20260926 06:15:23.195524 8566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:15:23.196768 8634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:23.199396 8634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644: Bootstrap starting.
I20260926 06:15:23.200006 8634 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:23.200278 8634 log.cc:824] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644: Log is configured to *not* fsync() on all Append() calls
I20260926 06:15:23.200521 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 8566
I20260926 06:15:23.200644 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-0/wal/instance
I20260926 06:15:23.200858 2491 external_mini_cluster.cc:1428] Setting key dd288d1e33da881dc39d6eba4a451746
I20260926 06:15:23.201267 8634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644: No bootstrap required, opened a new log
I20260926 06:15:23.203696 8634 raft_consensus.cc:348] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } }
I20260926 06:15:23.203905 8634 raft_consensus.cc:374] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:23.203956 8634 raft_consensus.cc:729] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eed8ee75c5394469994882db1a963644, State: Initialized, Role: FOLLOWER
I20260926 06:15:23.204144 8634 consensus_queue.cc:260] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } }
I20260926 06:15:23.204228 8634 raft_consensus.cc:388] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:15:23.204262 8634 raft_consensus.cc:482] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:15:23.204315 8634 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:23.205119 8634 raft_consensus.cc:504] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } }
I20260926 06:15:23.205240 8634 leader_election.cc:302] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eed8ee75c5394469994882db1a963644; no voters:
I20260926 06:15:23.205435 8634 leader_election.cc:288] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:15:23.205513 8639 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:15:23.205699 8639 raft_consensus.cc:686] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644 [term 1 LEADER]: Becoming Leader. State: Replica: eed8ee75c5394469994882db1a963644, State: Running, Role: LEADER
I20260926 06:15:23.205754 8634 sys_catalog.cc:564] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:15:23.205798 8639 consensus_queue.cc:237] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } }
I20260926 06:15:23.206142 8640 sys_catalog.cc:455] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eed8ee75c5394469994882db1a963644" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } }
I20260926 06:15:23.206270 8640 sys_catalog.cc:458] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644 [sys.catalog]: This master's current role is: LEADER
I20260926 06:15:23.206514 8646 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:15:23.206471 8641 sys_catalog.cc:455] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eed8ee75c5394469994882db1a963644. Latest consensus state: current_term: 1 leader_uuid: "eed8ee75c5394469994882db1a963644" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } }
I20260926 06:15:23.206619 8641 sys_catalog.cc:458] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644 [sys.catalog]: This master's current role is: LEADER
I20260926 06:15:23.207479 8646 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:15:23.209355 8646 catalog_manager.cc:1347] Generated new cluster ID: b50e97be6eac4bccbae81dfbad9212a1
I20260926 06:15:23.209415 8646 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 06:15:23 dist-test-slave-hndd krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM
I20260926 06:15:23.214417 8638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:23.201555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49274 (local address 127.2.110.254:46749)
0926 06:15:23.201740 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:23.201751 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:23.201831 (+ 80us) server_negotiation.cc:407] Connection header received
0926 06:15:23.202781 (+ 950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:23.202791 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:23.202873 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:23.203052 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:23.203674 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:23.205558 (+ 1884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:23.206908 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:23.207523 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:23.208191 (+ 668us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:23.210553 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:23.210558 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:23.210558 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:23.210570 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:23.212661 (+ 2091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:23.213062 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:23.213065 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:23.213065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:23.213177 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:23.213441 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:23.213442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:23.213442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:23.213766 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:23.213930 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:23.214053 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 06:15:23.214199 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":76,"threads_started":1}
I20260926 06:15:23.219501 8646 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:15:23.220115 8646 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:15:23.227007 8646 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644: Generated new TSK 0
I20260926 06:15:23.227187 8646 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/oryx_127.2.110.193.keytab.
Entry for principal oryx/127.2.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/oryx_127.2.110.193.keytab.
I20260926 06:15:23.260812 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.193:0
--local_ip_for_outbound_sockets=127.2.110.193
--webserver_interface=127.2.110.193
--webserver_port=0
--tserver_master_addrs=127.2.110.254:46749
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/oryx_127.2.110.193.keytab
--principal=oryx/127.2.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:41555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:15:23.352823 8660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:23.353060 8660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:23.353086 8660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:23.354354 8660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:23.354404 8660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:23.354434 8660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.193
I20260926 06:15:23.355978 8660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:41555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/oryx_127.2.110.193.keytab
--principal=oryx/127.2.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.110.193
--webserver_port=0
--tserver_master_addrs=127.2.110.254:46749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.193
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:23.356240 8660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:23.356536 8660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:23.362705 8666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:23.362884 8668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:23.362948 8660 server_base.cc:1034] running on GCE node
W20260926 06:15:23.362972 8665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:23.363371 8660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:23.363687 8660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:23.364861 8660 hybrid_clock.cc:648] HybridClock initialized: now 1790403323364846 us; error 38 us; skew 500 ppm
Sep 26 06:15:23 dist-test-slave-hndd krb5kdc[8553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403323, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:23.367806 8660 init.cc:377] Logged in from keytab as oryx/127.2.110.193@KRBTEST.COM (short username oryx)
I20260926 06:15:23.368291 8660 webserver.cc:460] Webserver started at http://127.2.110.193:40447/ using document root <none> and password file <none>
I20260926 06:15:23.368530 8660 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:23.368602 8660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:23.368760 8660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:23.369954 8660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-0/data/instance:
uuid: "8217ae7c8e30427aa59ee96d67f4bf91"
format_stamp: "Formatted at 2026-09-26 06:15:23 on dist-test-slave-hndd"
server_key: "51fb783c79dd99c5f68b58c717542e01"
server_key_iv: "ec67182dac02997f2140a9cbf153bfec"
server_key_version: "encryptionkey@0"
I20260926 06:15:23.370338 8660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-0/wal/instance:
uuid: "8217ae7c8e30427aa59ee96d67f4bf91"
format_stamp: "Formatted at 2026-09-26 06:15:23 on dist-test-slave-hndd"
server_key: "51fb783c79dd99c5f68b58c717542e01"
server_key_iv: "ec67182dac02997f2140a9cbf153bfec"
server_key_version: "encryptionkey@0"
I20260926 06:15:23.371959 8660 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:15:23.373028 8675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:23.373328 8660 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:23.373492 8660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-0/wal
uuid: "8217ae7c8e30427aa59ee96d67f4bf91"
format_stamp: "Formatted at 2026-09-26 06:15:23 on dist-test-slave-hndd"
server_key: "51fb783c79dd99c5f68b58c717542e01"
server_key_iv: "ec67182dac02997f2140a9cbf153bfec"
server_key_version: "encryptionkey@0"
I20260926 06:15:23.373586 8660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:23.385452 8660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:23.388458 8660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:23.388664 8660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:23.389094 8660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:15:23.389532 8660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:15:23.389591 8660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:23.389628 8660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:15:23.389678 8660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:23.397109 8660 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.193:40801
I20260926 06:15:23.397149 8788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.193:40801 every 8 connection(s)
I20260926 06:15:23.397563 8660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
Sep 26 06:15:23 dist-test-slave-hndd krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403323, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.110.193@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM
I20260926 06:15:23.407346 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 8660
I20260926 06:15:23.407464 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-0/wal/instance
I20260926 06:15:23.407665 2491 external_mini_cluster.cc:1428] Setting key 7bd1521653f7b3efdca172ed3d7e042b
I20260926 06:15:23.408843 8638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:23.398528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:39303 (local address 127.2.110.254:46749)
0926 06:15:23.398574 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:23.398577 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:23.398779 (+ 202us) server_negotiation.cc:407] Connection header received
0926 06:15:23.399342 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:23.399343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:23.399357 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:23.399406 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:23.400415 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:23.400951 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:23.402085 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:23.402304 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:23.402392 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:23.404803 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:23.404804 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:23.404804 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:23.404812 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:23.406483 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:23.407013 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:23.407014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:23.407015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:23.407077 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:23.407486 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:23.407487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:23.407488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:23.407647 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:23.407944 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:23.408532 (+ 588us) server_negotiation.cc:299] Negotiation successful
0926 06:15:23.408634 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 06:15:23.409053 8791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:23.398484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:46749 (local address 127.2.110.193:39303)
0926 06:15:23.398635 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:23.398682 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:23.399218 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:23.399486 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:23.399490 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:23.400094 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:23.400294 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:23.400298 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:23.401056 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:23.401057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:23.401969 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:23.401972 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:23.402400 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:23.402401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:23.402533 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:23.403130 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:23.403139 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:23.404655 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:23.406582 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:23.406589 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:23.406591 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:23.406833 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:23.407237 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:23.407238 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:23.407238 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:23.407392 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:23.407921 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:23.407939 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:23.408348 (+ 409us) client_negotiation.cc:769] Sending connection context
0926 06:15:23.408490 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 06:15:23.408672 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":69,"threads_started":1}
I20260926 06:15:23.409581 8789 heartbeater.cc:347] Connected to a master server at 127.2.110.254:46749
I20260926 06:15:23.409669 8789 heartbeater.cc:464] Registering TS with master...
I20260926 06:15:23.409878 8789 heartbeater.cc:511] Master 127.2.110.254:46749 requested a full tablet report, sending...
I20260926 06:15:23.410454 8598 ts_manager.cc:194] Registered new tserver with Master: 8217ae7c8e30427aa59ee96d67f4bf91 (127.2.110.193:40801)
I20260926 06:15:23.411103 8598 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.110.193@KRBTEST.COM'} at 127.2.110.193:39303
WARNING: no policy specified for oryx/127.2.110.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.110.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/oryx_127.2.110.194.keytab.
Entry for principal oryx/127.2.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/oryx_127.2.110.194.keytab.
I20260926 06:15:23.436142 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.194:0
--local_ip_for_outbound_sockets=127.2.110.194
--webserver_interface=127.2.110.194
--webserver_port=0
--tserver_master_addrs=127.2.110.254:46749
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/oryx_127.2.110.194.keytab
--principal=oryx/127.2.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:41555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:15:23.528074 8794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:23.528312 8794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:23.528338 8794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:23.529620 8794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:23.529675 8794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:23.529706 8794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.194
I20260926 06:15:23.531235 8794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:41555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/oryx_127.2.110.194.keytab
--principal=oryx/127.2.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.110.194
--webserver_port=0
--tserver_master_addrs=127.2.110.254:46749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.194
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:23.531544 8794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:23.531935 8794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:23.537210 8800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:23.537184 8799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:23.537242 8802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:23.537633 8794 server_base.cc:1034] running on GCE node
I20260926 06:15:23.537863 8794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:23.538084 8794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:23.539247 8794 hybrid_clock.cc:648] HybridClock initialized: now 1790403323539221 us; error 41 us; skew 500 ppm
Sep 26 06:15:23 dist-test-slave-hndd krb5kdc[8553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403323, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:23.541886 8794 init.cc:377] Logged in from keytab as oryx/127.2.110.194@KRBTEST.COM (short username oryx)
I20260926 06:15:23.542361 8794 webserver.cc:460] Webserver started at http://127.2.110.194:43489/ using document root <none> and password file <none>
I20260926 06:15:23.542559 8794 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:23.542603 8794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:23.542695 8794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:23.543737 8794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-1/data/instance:
uuid: "2f74a0910ea94f6aa814996cc6eff5be"
format_stamp: "Formatted at 2026-09-26 06:15:23 on dist-test-slave-hndd"
server_key: "87811499738e6a86ac561019b5fd7416"
server_key_iv: "c7dc60da6e563d1fec8491c67601451f"
server_key_version: "encryptionkey@0"
I20260926 06:15:23.544034 8794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-1/wal/instance:
uuid: "2f74a0910ea94f6aa814996cc6eff5be"
format_stamp: "Formatted at 2026-09-26 06:15:23 on dist-test-slave-hndd"
server_key: "87811499738e6a86ac561019b5fd7416"
server_key_iv: "c7dc60da6e563d1fec8491c67601451f"
server_key_version: "encryptionkey@0"
I20260926 06:15:23.545426 8794 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 06:15:23.546255 8809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:23.546535 8794 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:23.546640 8794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-1/wal
uuid: "2f74a0910ea94f6aa814996cc6eff5be"
format_stamp: "Formatted at 2026-09-26 06:15:23 on dist-test-slave-hndd"
server_key: "87811499738e6a86ac561019b5fd7416"
server_key_iv: "c7dc60da6e563d1fec8491c67601451f"
server_key_version: "encryptionkey@0"
I20260926 06:15:23.546696 8794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:23.560873 8794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:23.563419 8794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:23.563575 8794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:23.563840 8794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:15:23.564111 8794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:15:23.564143 8794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:23.564167 8794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:15:23.564178 8794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:23.570103 8794 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.194:45563
I20260926 06:15:23.570307 8922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.194:45563 every 8 connection(s)
I20260926 06:15:23.570569 8794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
I20260926 06:15:23.571816 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 8794
I20260926 06:15:23.571928 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-1/wal/instance
I20260926 06:15:23.572136 2491 external_mini_cluster.cc:1428] Setting key adab3eb359a440ac867c3a339fd75e3c
Sep 26 06:15:23 dist-test-slave-hndd krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403323, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.110.194@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.2.110.195@KRBTEST.COM; defaulting to no policy
I20260926 06:15:23.583067 8638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:23.571519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:37495 (local address 127.2.110.254:46749)
0926 06:15:23.571618 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:23.571621 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:23.572031 (+ 410us) server_negotiation.cc:407] Connection header received
0926 06:15:23.572764 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:23.572766 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:23.572781 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:23.572833 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:23.574140 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:23.574648 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:23.576700 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:23.576996 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:23.577074 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:23.579926 (+ 2852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:23.579927 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:23.579928 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:23.579939 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:23.581494 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:23.581965 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:23.581967 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:23.581967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:23.582031 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:23.582313 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:23.582314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:23.582314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:23.582451 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:23.582536 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:23.582911 (+ 375us) server_negotiation.cc:299] Negotiation successful
0926 06:15:23.582997 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 06:15:23.583214 8925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:23.571437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:46749 (local address 127.2.110.194:37495)
0926 06:15:23.571773 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:23.571852 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:23.572608 (+ 756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:23.572919 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:23.572923 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:23.573705 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:23.574003 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:23.574011 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:23.574924 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:23.574927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:23.576582 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:23.576587 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:23.577108 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:23.577110 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:23.577303 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:23.577940 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:23.577952 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:23.579784 (+ 1832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:23.581603 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:23.581612 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:23.581615 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:23.581866 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:23.582103 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:23.582104 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:23.582104 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:23.582225 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:23.582557 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:23.582564 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:23.582708 (+ 144us) client_negotiation.cc:769] Sending connection context
0926 06:15:23.582832 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 06:15:23.582984 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":114,"threads_started":1}
I20260926 06:15:23.583734 8923 heartbeater.cc:347] Connected to a master server at 127.2.110.254:46749
I20260926 06:15:23.583847 8923 heartbeater.cc:464] Registering TS with master...
I20260926 06:15:23.584069 8923 heartbeater.cc:511] Master 127.2.110.254:46749 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.110.195@KRBTEST.COM" created.
I20260926 06:15:23.584571 8598 ts_manager.cc:194] Registered new tserver with Master: 2f74a0910ea94f6aa814996cc6eff5be (127.2.110.194:45563)
I20260926 06:15:23.585167 8598 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.110.194@KRBTEST.COM'} at 127.2.110.194:37495
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/oryx_127.2.110.195.keytab.
Entry for principal oryx/127.2.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/oryx_127.2.110.195.keytab.
I20260926 06:15:23.601564 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.195:0
--local_ip_for_outbound_sockets=127.2.110.195
--webserver_interface=127.2.110.195
--webserver_port=0
--tserver_master_addrs=127.2.110.254:46749
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/oryx_127.2.110.195.keytab
--principal=oryx/127.2.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:41555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:15:23.691982 8928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:23.692472 8928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:23.692513 8928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:23.694783 8928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:23.694849 8928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:23.694883 8928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.195
I20260926 06:15:23.696728 8928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:41555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/oryx_127.2.110.195.keytab
--principal=oryx/127.2.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.110.195
--webserver_port=0
--tserver_master_addrs=127.2.110.254:46749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.195
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:23.696971 8928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:23.697178 8928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:23.702911 8933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:23.702881 8934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:23.702880 8936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:23.703166 8928 server_base.cc:1034] running on GCE node
I20260926 06:15:23.703436 8928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:23.703706 8928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:23.704897 8928 hybrid_clock.cc:648] HybridClock initialized: now 1790403323704888 us; error 41 us; skew 500 ppm
Sep 26 06:15:23 dist-test-slave-hndd krb5kdc[8553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403323, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:23.708056 8928 init.cc:377] Logged in from keytab as oryx/127.2.110.195@KRBTEST.COM (short username oryx)
I20260926 06:15:23.708487 8928 webserver.cc:460] Webserver started at http://127.2.110.195:33753/ using document root <none> and password file <none>
I20260926 06:15:23.708755 8928 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:23.708807 8928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:23.708973 8928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:23.710171 8928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-2/data/instance:
uuid: "96a7adaf68cd451ba1388e6550cf780c"
format_stamp: "Formatted at 2026-09-26 06:15:23 on dist-test-slave-hndd"
server_key: "3d2578838376e92c2d3393695df2c5b8"
server_key_iv: "808b459b1670427d4f45f3ac7b790217"
server_key_version: "encryptionkey@0"
I20260926 06:15:23.710621 8928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-2/wal/instance:
uuid: "96a7adaf68cd451ba1388e6550cf780c"
format_stamp: "Formatted at 2026-09-26 06:15:23 on dist-test-slave-hndd"
server_key: "3d2578838376e92c2d3393695df2c5b8"
server_key_iv: "808b459b1670427d4f45f3ac7b790217"
server_key_version: "encryptionkey@0"
I20260926 06:15:23.712126 8928 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 06:15:23.712996 8943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:23.713399 8928 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:15:23.713527 8928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-2/wal
uuid: "96a7adaf68cd451ba1388e6550cf780c"
format_stamp: "Formatted at 2026-09-26 06:15:23 on dist-test-slave-hndd"
server_key: "3d2578838376e92c2d3393695df2c5b8"
server_key_iv: "808b459b1670427d4f45f3ac7b790217"
server_key_version: "encryptionkey@0"
I20260926 06:15:23.713610 8928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:23.740058 8928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:23.742617 8928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:23.742806 8928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:23.743062 8928 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:15:23.743384 8928 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:15:23.743439 8928 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:23.743484 8928 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:15:23.743515 8928 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:23.749810 8928 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.195:38407
I20260926 06:15:23.749850 9056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.195:38407 every 8 connection(s)
I20260926 06:15:23.750201 8928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
Sep 26 06:15:23 dist-test-slave-hndd krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403323, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.110.195@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM
I20260926 06:15:23.757519 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 8928
I20260926 06:15:23.757615 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-2/wal/instance
I20260926 06:15:23.757809 2491 external_mini_cluster.cc:1428] Setting key 170f52a9a95cc3060719b94377d8ef92
I20260926 06:15:23.760560 8638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:23.750955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:36231 (local address 127.2.110.254:46749)
0926 06:15:23.751030 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:23.751032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:23.751214 (+ 182us) server_negotiation.cc:407] Connection header received
0926 06:15:23.751709 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:23.751711 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:23.751722 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:23.751787 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:23.752738 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:23.753272 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:23.754325 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:23.754514 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:23.754570 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:23.756840 (+ 2270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:23.756841 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:23.756841 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:23.756848 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:23.758653 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:23.759441 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:23.759442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:23.759442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:23.759502 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:23.759806 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:23.759807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:23.759807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:23.759945 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:23.760018 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:23.760410 (+ 392us) server_negotiation.cc:299] Negotiation successful
0926 06:15:23.760494 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 06:15:23.760802 9059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:23.750934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:46749 (local address 127.2.110.195:36231)
0926 06:15:23.751085 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:23.751127 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:23.751601 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:23.751839 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:23.751843 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:23.752427 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:23.752625 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:23.752630 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:23.753446 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:23.753447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:23.754238 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:23.754241 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:23.754597 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:23.754597 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:23.754709 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:23.755335 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:23.755343 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:23.756721 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:23.758894 (+ 2173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:23.758910 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:23.758914 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:23.759312 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:23.759577 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:23.759578 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:23.759578 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:23.759700 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:23.760074 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:23.760084 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:23.760255 (+ 171us) client_negotiation.cc:769] Sending connection context
0926 06:15:23.760373 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 06:15:23.760540 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":82,"threads_started":1}
I20260926 06:15:23.761315 9057 heartbeater.cc:347] Connected to a master server at 127.2.110.254:46749
I20260926 06:15:23.761475 9057 heartbeater.cc:464] Registering TS with master...
I20260926 06:15:23.761641 9057 heartbeater.cc:511] Master 127.2.110.254:46749 requested a full tablet report, sending...
I20260926 06:15:23.761996 8598 ts_manager.cc:194] Registered new tserver with Master: 96a7adaf68cd451ba1388e6550cf780c (127.2.110.195:38407)
I20260926 06:15:23.762362 8598 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.110.195@KRBTEST.COM'} at 127.2.110.195:36231
I20260926 06:15:23.769898 2491 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.110.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.110.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.110.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/oryx_127.2.110.253.keytab.
Entry for principal oryx/127.2.110.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/oryx_127.2.110.253.keytab.
I20260926 06:15:23.797730 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.253:35513
--webserver_interface=127.2.110.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/oryx_127.2.110.253.keytab
--principal=oryx/127.2.110.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:41555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.110.254:46749,127.2.110.253:35513
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:15:23.884321 9062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:23.884572 9062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:23.884611 9062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:23.885849 9062 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:15:23.885919 9062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:23.885944 9062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:23.885965 9062 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:15:23.885995 9062 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:15:23.887529 9062 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:41555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.110.254:46749,127.2.110.253:35513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/oryx_127.2.110.253.keytab
--principal=oryx/127.2.110.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.253:35513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.110.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
W20260926 06:15:23.887677 9062 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.2.110.254:46749,127.2.110.253:35513'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 06:15:23.887843 9062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:23.888075 9062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:23.893308 9068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:23.893362 9067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:23.893322 9070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:23.893594 9062 server_base.cc:1034] running on GCE node
I20260926 06:15:23.893832 9062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:23.894153 9062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:23.895332 9062 hybrid_clock.cc:648] HybridClock initialized: now 1790403323895312 us; error 28 us; skew 500 ppm
Sep 26 06:15:23 dist-test-slave-hndd krb5kdc[8553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403323, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.110.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:23.898140 9062 init.cc:377] Logged in from keytab as oryx/127.2.110.253@KRBTEST.COM (short username oryx)
I20260926 06:15:23.898707 9062 webserver.cc:460] Webserver started at http://127.2.110.253:34361/ using document root <none> and password file <none>
I20260926 06:15:23.899008 9062 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:23.899061 9062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:23.899158 9062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:23.900609 9062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-1/data/instance:
uuid: "4a5a9e72e816463b9cfcb45b2f858088"
format_stamp: "Formatted at 2026-09-26 06:15:23 on dist-test-slave-hndd"
server_key: "470e99887769527dd4f465b29d5abae1"
server_key_iv: "32c6ba22686df31ae70a1a4814248a65"
server_key_version: "encryptionkey@0"
I20260926 06:15:23.900933 9062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-1/wal/instance:
uuid: "4a5a9e72e816463b9cfcb45b2f858088"
format_stamp: "Formatted at 2026-09-26 06:15:23 on dist-test-slave-hndd"
server_key: "470e99887769527dd4f465b29d5abae1"
server_key_iv: "32c6ba22686df31ae70a1a4814248a65"
server_key_version: "encryptionkey@0"
I20260926 06:15:23.902479 9062 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:15:23.903374 9077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:23.903610 9062 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:15:23.903702 9062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-1/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-1/wal
uuid: "4a5a9e72e816463b9cfcb45b2f858088"
format_stamp: "Formatted at 2026-09-26 06:15:23 on dist-test-slave-hndd"
server_key: "470e99887769527dd4f465b29d5abae1"
server_key_iv: "32c6ba22686df31ae70a1a4814248a65"
server_key_version: "encryptionkey@0"
I20260926 06:15:23.903762 9062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:23.931844 9062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:23.934128 9062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:23.934242 9062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:23.938441 9062 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.253:35513
I20260926 06:15:23.938474 9129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.253:35513 every 8 connection(s)
I20260926 06:15:23.938833 9062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-1/data/info.pb
I20260926 06:15:23.939885 9130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:23.941495 9130 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } has no permanent_uuid. Determining permanent_uuid...
I20260926 06:15:23.943361 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 9062
Sep 26 06:15:23 dist-test-slave-hndd krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403323, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.110.253@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM
Sep 26 06:15:23 dist-test-slave-hndd krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.110.253@KRBTEST.COM
I20260926 06:15:23.951901 8638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:23.942239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49288 (local address 127.2.110.254:46749)
0926 06:15:23.942320 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:23.942323 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:23.942558 (+ 235us) server_negotiation.cc:407] Connection header received
0926 06:15:23.943074 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:23.943075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:23.943087 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:23.943145 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:23.943969 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:23.944484 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:23.945555 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:23.945746 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:23.945836 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:23.948208 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:23.948210 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:23.948210 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:23.948220 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:23.950065 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:23.950625 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:23.950626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:23.950626 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:23.950690 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:23.950984 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:23.950985 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:23.950985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:23.951204 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:23.951286 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:23.951670 (+ 384us) server_negotiation.cc:299] Negotiation successful
0926 06:15:23.951778 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 06:15:23.952183 9132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:23.942257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:46749 (local address 127.0.0.1:49288)
0926 06:15:23.942430 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:23.942464 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:23.942966 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:23.943216 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:23.943219 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:23.943539 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:23.943819 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:23.943826 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:23.944606 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:23.944607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:23.945453 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:23.945455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:23.945859 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:23.945860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:23.945987 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:23.946575 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:23.946585 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:23.948121 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:23.950149 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:23.950154 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:23.950156 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:23.950544 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:23.950791 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:23.950792 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:23.950792 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:23.950904 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:23.951310 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:23.951315 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:23.951509 (+ 194us) client_negotiation.cc:769] Sending connection context
0926 06:15:23.951649 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 06:15:23.951815 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":91,"threads_started":1}
I20260926 06:15:23.952680 9134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:23.943886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54144 (local address 127.2.110.253:35513)
0926 06:15:23.944058 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:23.944064 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:23.944102 (+ 38us) server_negotiation.cc:407] Connection header received
0926 06:15:23.944153 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:23.944155 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:23.944174 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:23.944262 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:23.944830 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:23.945632 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:23.946432 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:23.946871 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:23.946957 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:23.949505 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:23.949513 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:23.949514 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:23.949521 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:23.951310 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:23.951734 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:23.951737 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:23.951737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:23.951797 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:23.952032 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:23.952033 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:23.952034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:23.952190 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:23.952358 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:23.952467 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 06:15:23.952556 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":110,"threads_started":1}
I20260926 06:15:23.952857 9130 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35513 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 06:15:23 dist-test-slave-hndd krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403323, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.110.253@KRBTEST.COM for oryx/127.2.110.253@KRBTEST.COM
I20260926 06:15:23.959298 9134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:23.953153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54154 (local address 127.2.110.253:35513)
0926 06:15:23.953183 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:23.953185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:23.953192 (+ 7us) server_negotiation.cc:407] Connection header received
0926 06:15:23.953259 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:23.953260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:23.953269 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:23.953323 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:23.953603 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:23.954078 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:23.954750 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:23.954949 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:23.955033 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:23.956873 (+ 1840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:23.956874 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:23.956875 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:23.956880 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:23.958112 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:23.958452 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:23.958453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:23.958454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:23.958561 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:23.958877 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:23.958878 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:23.958878 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:23.958992 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:23.959051 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:23.959163 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 06:15:23.959229 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":16}
I20260926 06:15:23.959466 9132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:23.953090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.253:35513 (local address 127.0.0.1:54154)
0926 06:15:23.953144 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:23.953156 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:23.953203 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:23.953326 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:23.953327 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:23.953478 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:23.953520 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:23.953521 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:23.954171 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:23.954172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:23.954653 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:23.954654 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:23.955031 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:23.955032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:23.955097 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:23.955532 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:23.955537 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:23.956778 (+ 1241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:23.958182 (+ 1404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:23.958183 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:23.958183 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:23.958344 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:23.958659 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:23.958660 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:23.958660 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:23.958758 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:23.959056 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:23.959057 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:23.959095 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 06:15:23.959147 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 06:15:23.959297 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":66}
I20260926 06:15:23.960969 9130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088: Bootstrap starting.
I20260926 06:15:23.961998 9130 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:23.962324 9130 log.cc:824] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088: Log is configured to *not* fsync() on all Append() calls
I20260926 06:15:23.963117 9130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088: No bootstrap required, opened a new log
I20260926 06:15:23.964818 9130 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } peers { permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35513 } }
I20260926 06:15:23.964948 9130 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:23.964972 9130 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a5a9e72e816463b9cfcb45b2f858088, State: Initialized, Role: FOLLOWER
I20260926 06:15:23.965129 9130 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } peers { permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35513 } }
I20260926 06:15:23.965487 9130 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:15:23.965478 9136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } peers { permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35513 } } }
I20260926 06:15:23.965619 9136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 06:15:23.967275 9147 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 06:15:23.967337 9147 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 06:15:23.968993 9062 master_runner.cc:428] Detected that this master 4a5a9e72e816463b9cfcb45b2f858088 is joining an existing cluster
I20260926 06:15:23.969053 9062 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.110.253:35513
I20260926 06:15:23.977774 9153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:23.970079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:46749 (local address 127.0.0.1:49302)
0926 06:15:23.970655 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:23.970665 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:23.970733 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:23.970851 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:23.970851 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:23.971049 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:23.971106 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:23.971107 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:23.971782 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:23.971783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:23.972361 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:23.972363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:23.972750 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:23.972751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:23.972812 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:23.973383 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:23.973388 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:23.973897 (+ 509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:23.976043 (+ 2146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:23.976044 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:23.976045 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:23.976731 (+ 686us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:23.976966 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:23.976967 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:23.976967 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:23.977043 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:23.977405 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:23.977405 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:23.977459 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 06:15:23.977541 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 06:15:23.977655 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"mutex_wait_us":2,"thread_start_us":89,"threads_started":1}
I20260926 06:15:23.977842 8638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:23.970188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49302 (local address 127.2.110.254:46749)
0926 06:15:23.970300 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:23.970302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:23.970711 (+ 409us) server_negotiation.cc:407] Connection header received
0926 06:15:23.970793 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:23.970794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:23.970804 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:23.970850 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:23.971185 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:23.971695 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:23.972451 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:23.972663 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:23.972743 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:23.974013 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:23.974014 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:23.974015 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:23.974023 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:23.975946 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:23.976823 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:23.976824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:23.976825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:23.976893 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:23.977143 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:23.977144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:23.977144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:23.977306 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:23.977418 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:23.977617 (+ 199us) server_negotiation.cc:299] Negotiation successful
0926 06:15:23.977704 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 06:15:23.978246 9134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:23.970243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54168 (local address 127.2.110.253:35513)
0926 06:15:23.970326 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:23.970328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:23.970534 (+ 206us) server_negotiation.cc:407] Connection header received
0926 06:15:23.970608 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:23.970609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:23.970620 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:23.970683 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:23.971203 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:23.972004 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:23.973175 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:23.973501 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:23.973596 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:23.974535 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:23.974537 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:23.974538 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:23.974545 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:23.976745 (+ 2200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:23.977175 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:23.977175 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:23.977176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:23.977217 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:23.977445 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:23.977445 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:23.977446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:23.977588 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:23.977649 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:23.978090 (+ 441us) server_negotiation.cc:299] Negotiation successful
0926 06:15:23.978166 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 06:15:23.978035 9154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:23.970079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.253:35513 (local address 127.0.0.1:54168)
0926 06:15:23.970466 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:23.970475 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:23.970550 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:23.970710 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:23.970711 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:23.970995 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:23.971099 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:23.971101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:23.972104 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:23.972105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:23.973074 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:23.973077 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:23.973598 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:23.973599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:23.973660 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:23.974073 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:23.974078 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:23.974456 (+ 378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:23.976852 (+ 2396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:23.976853 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:23.976853 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:23.977104 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:23.977274 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:23.977274 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:23.977274 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:23.977381 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:23.977669 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:23.977670 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:23.977737 (+ 67us) client_negotiation.cc:769] Sending connection context
0926 06:15:23.977798 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 06:15:23.977921 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":88,"threads_started":1}
Sep 26 06:15:23 dist-test-slave-hndd krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403323, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.110.254@KRBTEST.COM for oryx/127.2.110.253@KRBTEST.COM
I20260926 06:15:23.989037 9134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:23.980880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54184 (local address 127.2.110.253:35513)
0926 06:15:23.980930 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:23.980932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:23.981079 (+ 147us) server_negotiation.cc:407] Connection header received
0926 06:15:23.981164 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:23.981165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:23.981173 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:23.981219 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:23.981778 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:23.982321 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:23.983544 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:23.983765 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:23.983826 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:23.986339 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:23.986341 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:23.986341 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:23.986349 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:23.987779 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:23.988199 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:23.988200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:23.988201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:23.988260 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:23.988468 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:23.988469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:23.988469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:23.988587 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:23.988662 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:23.988849 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 06:15:23.988936 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 06:15:23.989063 9155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:23.980805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.253:35513 (local address 127.0.0.1:54184)
0926 06:15:23.980965 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:23.981009 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:23.981090 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:23.981224 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:23.981227 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:23.981613 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:23.981678 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:23.981680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:23.982442 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:23.982443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:23.983426 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:23.983429 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:23.983838 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:23.983839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:23.983928 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:23.984399 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:23.984404 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:23.986233 (+ 1829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:23.987889 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:23.987897 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:23.987900 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:23.988117 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:23.988328 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:23.988329 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:23.988329 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:23.988397 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:23.988655 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:23.988663 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:23.988721 (+ 58us) client_negotiation.cc:769] Sending connection context
0926 06:15:23.988826 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 06:15:23.988952 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":98,"threads_started":1}
I20260926 06:15:23.989959 8598 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.110.253:35513: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088"
member_type: NON_VOTER
last_known_addr {
host: "127.2.110.253"
port: 35513
}
attrs {
promote: true
}
}
dest_uuid: "eed8ee75c5394469994882db1a963644"
cas_config_opid_index: -1
I20260926 06:15:23.990283 8598 consensus_queue.cc:237] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } peers { permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: NON_VOTER last_known_addr { host: "127.2.110.253" port: 35513 } attrs { promote: true } }
I20260926 06:15:23.990792 9157 raft_consensus.cc:2932] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 4a5a9e72e816463b9cfcb45b2f858088 (127.2.110.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } peers { permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: NON_VOTER last_known_addr { host: "127.2.110.253" port: 35513 } attrs { promote: true } } }
I20260926 06:15:23.991072 9104 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:23.991962 9157 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.110.253:35513
I20260926 06:15:23.992002 9104 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088 [term 1 FOLLOWER]: Refusing update from remote peer eed8ee75c5394469994882db1a963644: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 06:15:23.992326 9062 master.cc:537] Master@127.2.110.253:35513 shutting down...
I20260926 06:15:23.992290 9156 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: NON_VOTER last_known_addr { host: "127.2.110.253" port: 35513 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:15:23.992341 9158 sys_catalog.cc:455] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "eed8ee75c5394469994882db1a963644" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } peers { permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: NON_VOTER last_known_addr { host: "127.2.110.253" port: 35513 } attrs { promote: true } } }
I20260926 06:15:23.992430 9158 sys_catalog.cc:458] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644 [sys.catalog]: This master's current role is: LEADER
W20260926 06:15:23.992980 8584 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260926 06:15:23.993465 8584 consensus_peers.cc:487] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644 -> Peer 4a5a9e72e816463b9cfcb45b2f858088 (127.2.110.253:35513): Couldn't send request to peer 4a5a9e72e816463b9cfcb45b2f858088. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 06:15:23.995137 9062 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:15:23.995240 9062 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:15:23.995296 9062 tablet_replica.cc:330] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088: stopping tablet replica
I20260926 06:15:24.008194 9062 master.cc:552] Master@127.2.110.253:35513 shutdown complete.
I20260926 06:15:24.008267 9062 master_runner.cc:305] Clearing existing system tablet
I20260926 06:15:24.008425 9062 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:24.008477 9062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:24.009270 9062 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260926 06:15:24.009912 9162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:24.010192 9062 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:24.010277 9062 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 06:15:24.010308 9062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-1/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-1/wal
uuid: "4a5a9e72e816463b9cfcb45b2f858088"
format_stamp: "Formatted at 2026-09-26 06:15:23 on dist-test-slave-hndd"
server_key: "470e99887769527dd4f465b29d5abae1"
server_key_iv: "32c6ba22686df31ae70a1a4814248a65"
server_key_version: "encryptionkey@0"
I20260926 06:15:24.010955 9062 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:15:24.012537 9062 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 06:15:24.012626 9062 log.cc:1196] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 06:15:24.012827 9062 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088: Deleting consensus metadata
I20260926 06:15:24.012972 9062 master_runner.cc:315] Copying system tablet from 127.2.110.254:46749
I20260926 06:15:24.013643 9062 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088: tablet copy: Beginning tablet copy session from remote peer at address 127.2.110.254:46749
I20260926 06:15:24.019940 8638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.014056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49316 (local address 127.2.110.254:46749)
0926 06:15:24.014107 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:24.014110 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:24.014171 (+ 61us) server_negotiation.cc:407] Connection header received
0926 06:15:24.014264 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:24.014265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:24.014275 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:24.014362 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:24.014684 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.015179 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.015988 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.016187 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.016273 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.017349 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:24.017350 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:24.017351 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:24.017358 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:24.018791 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:24.019153 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:24.019154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:24.019155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:24.019212 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:24.019415 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:24.019416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:24.019417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:24.019533 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:24.019611 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:24.019763 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 06:15:24.019844 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 06:15:24.019971 9168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.013984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:46749 (local address 127.0.0.1:49316)
0926 06:15:24.014100 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:24.014113 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:24.014199 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:24.014359 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:24.014360 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:24.014543 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:24.014603 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:24.014605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.015266 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.015267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:24.015900 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:24.015902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.016276 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.016277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:24.016353 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.016917 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:24.016922 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:24.017229 (+ 307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:24.018888 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:24.018889 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:24.018890 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:24.019069 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:24.019287 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:24.019287 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:24.019288 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:24.019349 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:24.019602 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:24.019603 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:24.019648 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 06:15:24.019738 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 06:15:24.019845 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":69,"threads_started":1}
I20260926 06:15:24.020608 8618 tablet_copy_service.cc:140] P eed8ee75c5394469994882db1a963644: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 4a5a9e72e816463b9cfcb45b2f858088 ({username='oryx', principal='oryx/127.2.110.253@KRBTEST.COM'} at 127.0.0.1:49316)
I20260926 06:15:24.020694 8618 tablet_copy_service.cc:161] P eed8ee75c5394469994882db1a963644: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 4a5a9e72e816463b9cfcb45b2f858088 at {username='oryx', principal='oryx/127.2.110.253@KRBTEST.COM'} at 127.0.0.1:49316: session id = 4a5a9e72e816463b9cfcb45b2f858088-00000000000000000000000000000000
I20260926 06:15:24.021391 8618 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 06:15:24.022053 9062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 06:15:24.023015 9062 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088: tablet copy: Starting download of 0 data blocks...
I20260926 06:15:24.023166 9062 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088: tablet copy: Starting download of 1 WAL segments...
I20260926 06:15:24.024017 9062 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 06:15:24.024885 8618 tablet_copy_service.cc:342] P eed8ee75c5394469994882db1a963644: Request end of tablet copy session 4a5a9e72e816463b9cfcb45b2f858088-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.110.253@KRBTEST.COM'} at 127.0.0.1:49316
I20260926 06:15:24.024955 8618 tablet_copy_service.cc:433] P eed8ee75c5394469994882db1a963644: ending tablet copy session 4a5a9e72e816463b9cfcb45b2f858088-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 4a5a9e72e816463b9cfcb45b2f858088
I20260926 06:15:24.027462 9062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:24.028694 9169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:24.028689 9170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:24.028749 9172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:24.029489 9062 server_base.cc:1034] running on GCE node
I20260926 06:15:24.029587 9062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:24.029722 9062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:24.030855 9062 hybrid_clock.cc:648] HybridClock initialized: now 1790403324030851 us; error 37 us; skew 500 ppm
Sep 26 06:15:24 dist-test-slave-hndd krb5kdc[8553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403324, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.110.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:24.032250 9062 init.cc:377] Logged in from keytab as oryx/127.2.110.253@KRBTEST.COM (short username oryx)
I20260926 06:15:24.032541 9062 webserver.cc:460] Webserver started at http://127.2.110.253:40617/ using document root <none> and password file <none>
I20260926 06:15:24.032670 9062 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:24.032716 9062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:24.033371 9062 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:24.033902 9179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:24.034060 9062 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 06:15:24.034121 9062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-1/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-1/wal
uuid: "4a5a9e72e816463b9cfcb45b2f858088"
format_stamp: "Formatted at 2026-09-26 06:15:23 on dist-test-slave-hndd"
server_key: "470e99887769527dd4f465b29d5abae1"
server_key_iv: "32c6ba22686df31ae70a1a4814248a65"
server_key_version: "encryptionkey@0"
I20260926 06:15:24.034179 9062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:24.055150 9062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:24.056823 9062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:24.059932 9062 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.253:35513
I20260926 06:15:24.060150 9231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.253:35513 every 8 connection(s)
I20260926 06:15:24.060364 9062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-1/data/info.pb
I20260926 06:15:24.061081 9232 sys_catalog.cc:263] Verifying existing consensus state
I20260926 06:15:24.061578 9232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088: Bootstrap starting.
I20260926 06:15:24.063760 9232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 06:15:24.063994 9232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088: Bootstrap complete.
I20260926 06:15:24.064213 9232 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } peers { permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: NON_VOTER last_known_addr { host: "127.2.110.253" port: 35513 } attrs { promote: true } }
I20260926 06:15:24.064298 9232 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4a5a9e72e816463b9cfcb45b2f858088, State: Initialized, Role: LEARNER
I20260926 06:15:24.064348 9232 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } peers { permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: NON_VOTER last_known_addr { host: "127.2.110.253" port: 35513 } attrs { promote: true } }
I20260926 06:15:24.064544 9232 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:15:24.064538 9235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } peers { permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: NON_VOTER last_known_addr { host: "127.2.110.253" port: 35513 } attrs { promote: true } } }
I20260926 06:15:24.064617 9235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088 [sys.catalog]: This master's current role is: LEARNER
I20260926 06:15:24.065970 9246 catalog_manager.cc:1259] Loaded cluster ID: b50e97be6eac4bccbae81dfbad9212a1
I20260926 06:15:24.066036 9246 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088: loading cluster ID for follower catalog manager: success
I20260926 06:15:24.066984 9246 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088: acquiring CA information for follower catalog manager: success
I20260926 06:15:24.067260 9246 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 06:15:24.113684 9247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.104929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54226 (local address 127.2.110.253:35513)
0926 06:15:24.105145 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:24.105147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:24.105157 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:15:24.105181 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:24.105182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:24.105192 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:24.105278 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:24.106125 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.107075 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.108101 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.108312 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.108376 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.109978 (+ 1602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:24.109980 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:24.109980 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:24.110025 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:24.112152 (+ 2127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:24.112672 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:24.112674 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:24.112675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:24.112771 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:24.113071 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:24.113073 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:24.113073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:24.113217 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:24.113311 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:24.113477 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 06:15:24.113554 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":111,"threads_started":1}
WARNING: no policy specified for oryx/127.2.110.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.110.252@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.110.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/oryx_127.2.110.252.keytab.
Entry for principal oryx/127.2.110.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/oryx_127.2.110.252.keytab.
I20260926 06:15:24.140401 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.252:41235
--webserver_interface=127.2.110.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/oryx_127.2.110.252.keytab
--principal=oryx/127.2.110.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:41555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.110.254:46749,127.2.110.253:35513,127.2.110.252:41235
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:15:24.227037 9250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:24.227268 9250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:24.227293 9250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:24.228471 9250 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:15:24.228516 9250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:24.228531 9250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:24.228544 9250 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:15:24.228557 9250 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:15:24.230167 9250 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:41555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.110.254:46749,127.2.110.253:35513,127.2.110.252:41235
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/oryx_127.2.110.252.keytab
--principal=oryx/127.2.110.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.252:41235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.110.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:24.230388 9250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:24.230576 9250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:24.235833 9255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:24.235821 9258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:24.235882 9256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:24.236377 9250 server_base.cc:1034] running on GCE node
I20260926 06:15:24.236652 9250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:24.236982 9250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:24.238166 9250 hybrid_clock.cc:648] HybridClock initialized: now 1790403324238155 us; error 49 us; skew 500 ppm
Sep 26 06:15:24 dist-test-slave-hndd krb5kdc[8553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403324, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.110.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:24.240849 9250 init.cc:377] Logged in from keytab as oryx/127.2.110.252@KRBTEST.COM (short username oryx)
I20260926 06:15:24.241317 9250 webserver.cc:460] Webserver started at http://127.2.110.252:38371/ using document root <none> and password file <none>
I20260926 06:15:24.241572 9250 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:24.241616 9250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:24.241708 9250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:24.242766 9250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-2/data/instance:
uuid: "f5662db075fb4054a7d7380e5834eb1f"
format_stamp: "Formatted at 2026-09-26 06:15:24 on dist-test-slave-hndd"
server_key: "becafeb546e706c72ac63128f480ad70"
server_key_iv: "e8b9a305279b1c7293af2b6d9650ce5a"
server_key_version: "encryptionkey@0"
I20260926 06:15:24.243075 9250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-2/wal/instance:
uuid: "f5662db075fb4054a7d7380e5834eb1f"
format_stamp: "Formatted at 2026-09-26 06:15:24 on dist-test-slave-hndd"
server_key: "becafeb546e706c72ac63128f480ad70"
server_key_iv: "e8b9a305279b1c7293af2b6d9650ce5a"
server_key_version: "encryptionkey@0"
I20260926 06:15:24.244364 9250 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:15:24.245129 9265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:24.245373 9250 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:15:24.245476 9250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-2/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-2/wal
uuid: "f5662db075fb4054a7d7380e5834eb1f"
format_stamp: "Formatted at 2026-09-26 06:15:24 on dist-test-slave-hndd"
server_key: "becafeb546e706c72ac63128f480ad70"
server_key_iv: "e8b9a305279b1c7293af2b6d9650ce5a"
server_key_version: "encryptionkey@0"
I20260926 06:15:24.245538 9250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:24.262411 9250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:24.265101 9250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:24.265272 9250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:24.269986 9250 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.252:41235
I20260926 06:15:24.270038 9317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.252:41235 every 8 connection(s)
I20260926 06:15:24.270428 9250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-2/data/info.pb
I20260926 06:15:24.271752 9318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:24.274008 9318 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } has no permanent_uuid. Determining permanent_uuid...
I20260926 06:15:24.276036 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 9250
Sep 26 06:15:24 dist-test-slave-hndd krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403324, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.110.252@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM
Sep 26 06:15:24 dist-test-slave-hndd krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.110.252@KRBTEST.COM
I20260926 06:15:24.286247 8638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.275162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49318 (local address 127.2.110.254:46749)
0926 06:15:24.275236 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:24.275240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:24.275827 (+ 587us) server_negotiation.cc:407] Connection header received
0926 06:15:24.276639 (+ 812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:24.276641 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:24.276654 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:24.276726 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:24.277475 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.278206 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.279608 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.279918 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.279981 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.282913 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:24.282915 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:24.282915 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:24.282928 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:24.284536 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:24.284992 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:24.284993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:24.284994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:24.285057 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:24.285327 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:24.285328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:24.285328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:24.285515 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:24.285571 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:24.285952 (+ 381us) server_negotiation.cc:299] Negotiation successful
0926 06:15:24.286155 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 06:15:24.286316 9320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.275022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:46749 (local address 127.0.0.1:49318)
0926 06:15:24.275232 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:24.275278 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:24.276365 (+ 1087us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:24.276832 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:24.276837 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:24.277130 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:24.277324 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:24.277328 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.278314 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.278315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:24.279502 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:24.279504 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.280015 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.280016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:24.280153 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.280987 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:24.280995 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:24.282748 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:24.284641 (+ 1893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:24.284647 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:24.284649 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:24.284906 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:24.285152 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:24.285153 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:24.285153 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:24.285256 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:24.285589 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:24.285595 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:24.285781 (+ 186us) client_negotiation.cc:769] Sending connection context
0926 06:15:24.285923 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 06:15:24.286109 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":208,"thread_start_us":124,"threads_started":1}
I20260926 06:15:24.286990 9318 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35513 } has no permanent_uuid. Determining permanent_uuid...
I20260926 06:15:24.287164 9321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.276661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35264 (local address 127.2.110.252:41235)
0926 06:15:24.276939 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:24.276943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:24.276960 (+ 17us) server_negotiation.cc:407] Connection header received
0926 06:15:24.276977 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:24.276979 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:24.276998 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:24.277094 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:24.277860 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.279065 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.279913 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.280343 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.280432 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.283392 (+ 2960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:24.283401 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:24.283401 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:24.283410 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:24.285316 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:24.285767 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:24.285770 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:24.285770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:24.285821 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:24.286131 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:24.286131 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:24.286132 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:24.286306 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:24.286892 (+ 586us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:24.286932 (+ 40us) server_negotiation.cc:299] Negotiation successful
0926 06:15:24.286994 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":238,"thread_start_us":150,"threads_started":1}
Sep 26 06:15:24 dist-test-slave-hndd krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403324, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.110.252@KRBTEST.COM for oryx/127.2.110.253@KRBTEST.COM
I20260926 06:15:24.295433 9320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.287282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.253:35513 (local address 127.0.0.1:54240)
0926 06:15:24.287519 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:24.287527 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:24.287558 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:24.287926 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:24.287928 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:24.288267 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:24.288329 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:24.288332 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.289218 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.289226 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:24.289919 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:24.289922 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.290331 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.290332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:24.290398 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.290922 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:24.290928 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:24.292210 (+ 1282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:24.294334 (+ 2124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:24.294335 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:24.294336 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:24.294526 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:24.294750 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:24.294751 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:24.294751 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:24.294810 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:24.295077 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:24.295078 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:24.295126 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 06:15:24.295215 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 06:15:24.295330 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222}
I20260926 06:15:24.295433 9247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.287542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54240 (local address 127.2.110.253:35513)
0926 06:15:24.287614 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:24.287616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:24.287625 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:15:24.287648 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:24.287649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:24.287661 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:24.287894 (+ 233us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:24.288440 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.289078 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.290011 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.290236 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.290333 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.292324 (+ 1991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:24.292326 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:24.292326 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:24.292335 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:24.294059 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:24.294609 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:24.294610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:24.294611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:24.294678 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:24.294884 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:24.294885 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:24.294885 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:24.295007 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:24.295088 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:24.295242 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 06:15:24.295334 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 06:15:24.296250 9318 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.110.252" port: 41235 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 06:15:24 dist-test-slave-hndd krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403324, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.110.252@KRBTEST.COM for oryx/127.2.110.252@KRBTEST.COM
I20260926 06:15:24.303735 9320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.296588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.252:41235 (local address 127.0.0.1:35278)
0926 06:15:24.296639 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:24.296649 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:24.296695 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:24.296845 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:24.296846 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:24.296982 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:24.297013 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:24.297014 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.297753 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.297754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:24.298236 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:24.298237 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.298629 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.298630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:24.298676 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.299152 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:24.299159 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:24.300885 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:24.302838 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:24.302838 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:24.302839 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:24.303014 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:24.303225 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:24.303225 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:24.303225 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:24.303274 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:24.303493 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:24.303494 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:24.303530 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 06:15:24.303580 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 06:15:24.303664 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 06:15:24.303812 9321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.296692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35278 (local address 127.2.110.252:41235)
0926 06:15:24.296752 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:24.296754 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:24.296761 (+ 7us) server_negotiation.cc:407] Connection header received
0926 06:15:24.296777 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:24.296778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:24.296787 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:24.296836 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:24.297095 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.297654 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.298338 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.298545 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.298631 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.300994 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:24.300996 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:24.300996 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:24.301004 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:24.302697 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:24.303101 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:24.303101 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:24.303102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:24.303156 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:24.303331 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:24.303332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:24.303332 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:24.303438 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:24.303491 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:24.303614 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 06:15:24.303734 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":41}
I20260926 06:15:24.305249 9318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f: Bootstrap starting.
I20260926 06:15:24.306236 9318 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:24.306638 9318 log.cc:824] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f: Log is configured to *not* fsync() on all Append() calls
I20260926 06:15:24.307698 9318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f: No bootstrap required, opened a new log
I20260926 06:15:24.309362 9318 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } peers { permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35513 } } peers { permanent_uuid: "f5662db075fb4054a7d7380e5834eb1f" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 41235 } }
I20260926 06:15:24.309546 9318 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:24.309587 9318 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5662db075fb4054a7d7380e5834eb1f, State: Initialized, Role: FOLLOWER
I20260926 06:15:24.309723 9318 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } peers { permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35513 } } peers { permanent_uuid: "f5662db075fb4054a7d7380e5834eb1f" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 41235 } }
I20260926 06:15:24.310084 9318 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:15:24.310889 9323 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } peers { permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35513 } } peers { permanent_uuid: "f5662db075fb4054a7d7380e5834eb1f" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 41235 } } }
I20260926 06:15:24.311002 9323 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f [sys.catalog]: This master's current role is: FOLLOWER
W20260926 06:15:24.312430 9334 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 06:15:24.312490 9334 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 06:15:24.315618 9250 master_runner.cc:428] Detected that this master f5662db075fb4054a7d7380e5834eb1f is joining an existing cluster
I20260926 06:15:24.315688 9250 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.110.252:41235
I20260926 06:15:24.325659 9321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.317014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35280 (local address 127.2.110.252:41235)
0926 06:15:24.317045 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:24.317047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:24.317304 (+ 257us) server_negotiation.cc:407] Connection header received
0926 06:15:24.317428 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:24.317428 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:24.317437 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:24.317462 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:24.318080 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.318842 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.320111 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.320398 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.320439 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.322407 (+ 1968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:24.322408 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:24.322409 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:24.322416 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:24.324092 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:24.324450 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:24.324451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:24.324452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:24.324513 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:24.324918 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:24.324919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:24.324920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:24.325071 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:24.325157 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:24.325428 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 06:15:24.325526 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":18}
I20260926 06:15:24.325752 9342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.317081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.252:41235 (local address 127.0.0.1:35280)
0926 06:15:24.317258 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:24.317266 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:24.317322 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:24.317516 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:24.317517 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:24.317889 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:24.317990 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:24.317992 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.318932 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.318933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:24.320003 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:24.320006 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.320533 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.320534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:24.321048 (+ 514us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.321628 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:24.321637 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:24.322286 (+ 649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:24.324198 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:24.324199 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:24.324199 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:24.324371 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:24.324674 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:24.324676 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:24.324677 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:24.324799 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:24.325193 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:24.325194 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:24.325241 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 06:15:24.325397 (+ 156us) client_negotiation.cc:240] Negotiation successful
0926 06:15:24.325585 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":156,"thread_start_us":53,"threads_started":1}
I20260926 06:15:24.338976 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 8660
I20260926 06:15:24.343075 8638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.316843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49330 (local address 127.2.110.254:46749)
0926 06:15:24.316977 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:24.316980 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:24.317240 (+ 260us) server_negotiation.cc:407] Connection header received
0926 06:15:24.317367 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:24.317368 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:24.317381 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:24.317413 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:24.318086 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.318828 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.320114 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.320426 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.320462 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.338880 (+ 18418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:24.338883 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:24.338883 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:24.338893 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:24.341700 (+ 2807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:24.342154 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:24.342155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:24.342156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:24.342214 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:24.342438 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:24.342439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:24.342439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:24.342589 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:24.342635 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:24.342853 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 06:15:24.342945 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260926 06:15:24.343753 9247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.316912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54252 (local address 127.2.110.253:35513)
0926 06:15:24.316983 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:24.316985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:24.317122 (+ 137us) server_negotiation.cc:407] Connection header received
0926 06:15:24.317229 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:24.317230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:24.317241 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:24.317305 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:24.318096 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.318859 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.320126 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.320447 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.320492 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.339133 (+ 18641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:24.339135 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:24.339135 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:24.339144 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:24.342556 (+ 3412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:24.342963 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:24.342964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:24.342964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:24.343017 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:24.343207 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:24.343208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:24.343209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:24.343345 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:24.343384 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:24.343581 (+ 197us) server_negotiation.cc:299] Negotiation successful
0926 06:15:24.343660 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 06:15:24.343784 9340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.316744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:46749 (local address 127.0.0.1:49330)
0926 06:15:24.317184 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:24.317193 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:24.317265 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:24.317484 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:24.317484 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:24.317903 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:24.317990 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:24.317992 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.318932 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.318933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:24.320003 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:24.320006 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.320536 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.320536 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:24.320596 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.338074 (+ 17478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:24.338083 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:24.338744 (+ 661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:24.341805 (+ 3061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:24.341806 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:24.341806 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:24.342054 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:24.342289 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:24.342290 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:24.342290 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:24.342348 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:24.342683 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:24.342684 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:24.342734 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 06:15:24.343539 (+ 805us) client_negotiation.cc:240] Negotiation successful
0926 06:15:24.343662 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"mutex_wait_us":88,"thread_start_us":82,"threads_started":1}
I20260926 06:15:24.344492 9341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.316742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.253:35513 (local address 127.0.0.1:54252)
0926 06:15:24.317043 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:24.317055 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:24.317156 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:24.317489 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:24.317491 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:24.317896 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:24.317996 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:24.317998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.318961 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.318961 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:24.320016 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:24.320019 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.320571 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.320571 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:24.320638 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.338222 (+ 17584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:24.338229 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:24.338990 (+ 761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:24.342649 (+ 3659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:24.342650 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:24.342650 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:24.342863 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:24.343085 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:24.343086 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:24.343086 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:24.343139 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:24.343429 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:24.343429 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:24.343470 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 06:15:24.343903 (+ 433us) client_negotiation.cc:240] Negotiation successful
0926 06:15:24.344001 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":37,"spinlock_wait_cycles":896,"thread_start_us":97,"threads_started":1}
I20260926 06:15:24.345400 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.193:40801
--local_ip_for_outbound_sockets=127.2.110.193
--tserver_master_addrs=127.2.110.254:46749
--webserver_port=40447
--webserver_interface=127.2.110.193
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/oryx_127.2.110.193.keytab
--principal=oryx/127.2.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:41555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.110.254:46749,127.2.110.253:35513
--tserver_master_addrs=127.2.110.254:46749,127.2.110.253:35513,127.2.110.252:41235 with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 06:15:24 dist-test-slave-hndd krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403323, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.110.254@KRBTEST.COM for oryx/127.2.110.252@KRBTEST.COM
I20260926 06:15:24.356700 9155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.347330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.252:41235 (local address 127.0.0.1:35286)
0926 06:15:24.347412 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:24.347436 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:24.347478 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:24.347914 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:24.347916 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:24.348179 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:24.348264 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:24.348266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.349407 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.349408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:24.350334 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:24.350338 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.350919 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.350920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:24.350988 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.351649 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:24.351657 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:24.353430 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:24.355575 (+ 2145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:24.355576 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:24.355577 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:24.355800 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:24.355998 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:24.355999 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:24.355999 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:24.356054 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:24.356339 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:24.356341 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:24.356391 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 06:15:24.356463 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 06:15:24.356592 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 06:15:24.357100 9321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.347557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35286 (local address 127.2.110.252:41235)
0926 06:15:24.347635 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:24.347637 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:24.347650 (+ 13us) server_negotiation.cc:407] Connection header received
0926 06:15:24.347699 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:24.347700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:24.347713 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:24.347799 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:24.348358 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.349258 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.350477 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.350781 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.350843 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.353568 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:24.353570 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:24.353570 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:24.353579 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:24.355457 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:24.355878 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:24.355879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:24.355879 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:24.355932 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:24.356115 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:24.356115 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:24.356116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:24.356260 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:24.356315 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:24.356918 (+ 603us) server_negotiation.cc:299] Negotiation successful
0926 06:15:24.357004 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 06:15:24.357694 8598 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.110.252:41235: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f5662db075fb4054a7d7380e5834eb1f"
member_type: NON_VOTER
last_known_addr {
host: "127.2.110.252"
port: 41235
}
attrs {
promote: true
}
}
dest_uuid: "eed8ee75c5394469994882db1a963644"
cas_config_opid_index: 5
I20260926 06:15:24.357954 8598 consensus_queue.cc:237] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } peers { permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: NON_VOTER last_known_addr { host: "127.2.110.253" port: 35513 } attrs { promote: true } } peers { permanent_uuid: "f5662db075fb4054a7d7380e5834eb1f" member_type: NON_VOTER last_known_addr { host: "127.2.110.252" port: 41235 } attrs { promote: true } }
I20260926 06:15:24.358361 9158 raft_consensus.cc:2932] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f5662db075fb4054a7d7380e5834eb1f (127.2.110.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } peers { permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: NON_VOTER last_known_addr { host: "127.2.110.253" port: 35513 } attrs { promote: true } } peers { permanent_uuid: "f5662db075fb4054a7d7380e5834eb1f" member_type: NON_VOTER last_known_addr { host: "127.2.110.252" port: 41235 } attrs { promote: true } } }
I20260926 06:15:24.359467 9158 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.110.252:41235
I20260926 06:15:24.359543 9158 sys_catalog.cc:455] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "eed8ee75c5394469994882db1a963644" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } peers { permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: NON_VOTER last_known_addr { host: "127.2.110.253" port: 35513 } attrs { promote: true } } peers { permanent_uuid: "f5662db075fb4054a7d7380e5834eb1f" member_type: NON_VOTER last_known_addr { host: "127.2.110.252" port: 41235 } attrs { promote: true } } }
I20260926 06:15:24.359615 9158 sys_catalog.cc:458] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644 [sys.catalog]: This master's current role is: LEADER
I20260926 06:15:24.360080 9292 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:24.361127 9292 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f [term 1 FOLLOWER]: Refusing update from remote peer eed8ee75c5394469994882db1a963644: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 06:15:24.361562 9156 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5662db075fb4054a7d7380e5834eb1f" member_type: NON_VOTER last_known_addr { host: "127.2.110.252" port: 41235 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:15:24.361987 9250 master.cc:537] Master@127.2.110.252:41235 shutting down...
I20260926 06:15:24.362901 9292 raft_consensus.cc:858] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } peers { permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35513 } } peers { permanent_uuid: "f5662db075fb4054a7d7380e5834eb1f" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 41235 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } peers { permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: NON_VOTER last_known_addr { host: "127.2.110.253" port: 35513 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } peers { permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: NON_VOTER last_known_addr { host: "127.2.110.253" port: 35513 } attrs { promote: true } } peers { permanent_uuid: "f5662db075fb4054a7d7380e5834eb1f" member_type: NON_VOTER last_known_addr { host: "127.2.110.252" port: 41235 } attrs { promote: true } }
W20260926 06:15:24.362998 9292 raft_consensus.cc:1565] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } peers { permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35513 } } peers { permanent_uuid: "f5662db075fb4054a7d7380e5834eb1f" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 41235 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } peers { permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: NON_VOTER last_known_addr { host: "127.2.110.253" port: 35513 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } peers { permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: NON_VOTER last_known_addr { host: "127.2.110.253" port: 35513 } attrs { promote: true } } peers { permanent_uuid: "f5662db075fb4054a7d7380e5834eb1f" member_type: NON_VOTER last_known_addr { host: "127.2.110.252" port: 41235 } attrs { promote: true } }
I20260926 06:15:24.363471 9292 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 4a5a9e72e816463b9cfcb45b2f858088 (127.2.110.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } peers { permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: NON_VOTER last_known_addr { host: "127.2.110.253" port: 35513 } attrs { promote: true } } }
I20260926 06:15:24.365031 9323 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader eed8ee75c5394469994882db1a963644. Latest consensus state: current_term: 1 leader_uuid: "eed8ee75c5394469994882db1a963644" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } peers { permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: NON_VOTER last_known_addr { host: "127.2.110.253" port: 35513 } attrs { promote: true } } }
I20260926 06:15:24.365115 9323 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260926 06:15:24.365154 9323 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eed8ee75c5394469994882db1a963644" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } peers { permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: NON_VOTER last_known_addr { host: "127.2.110.253" port: 35513 } attrs { promote: true } } }
I20260926 06:15:24.365198 9323 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260926 06:15:24.365250 9323 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eed8ee75c5394469994882db1a963644" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } peers { permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: NON_VOTER last_known_addr { host: "127.2.110.253" port: 35513 } attrs { promote: true } } }
I20260926 06:15:24.365284 9323 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260926 06:15:24.366518 9247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.358479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54256 (local address 127.2.110.253:35513)
0926 06:15:24.359699 (+ 1220us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:24.359701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:24.359713 (+ 12us) server_negotiation.cc:407] Connection header received
0926 06:15:24.359741 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:24.359742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:24.359768 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:24.359858 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:15:24.360044 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.360820 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.364955 (+ 4135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.365698 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.365752 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.365824 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:24.366356 (+ 532us) server_negotiation.cc:299] Negotiation successful
0926 06:15:24.366411 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1194}
I20260926 06:15:24.366954 9155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.358367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.253:35513 (local address 127.0.0.1:54256)
0926 06:15:24.358645 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:24.358656 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:24.358693 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:24.359884 (+ 1191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:24.359885 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:24.359887 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:15:24.359942 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:24.359944 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.360926 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.360927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:24.364567 (+ 3640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:24.364571 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.366040 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.366041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:24.366215 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.366248 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 06:15:24.366786 (+ 538us) client_negotiation.cc:240] Negotiation successful
0926 06:15:24.366848 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259}
I20260926 06:15:24.367460 9206 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088 [term 1 LEARNER]: Refusing update from remote peer eed8ee75c5394469994882db1a963644: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 06:15:24.367690 9156 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: NON_VOTER last_known_addr { host: "127.2.110.253" port: 35513 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260926 06:15:24.367928 9156 sys_catalog.cc:455] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eed8ee75c5394469994882db1a963644" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } peers { permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: NON_VOTER last_known_addr { host: "127.2.110.253" port: 35513 } attrs { promote: true } } peers { permanent_uuid: "f5662db075fb4054a7d7380e5834eb1f" member_type: NON_VOTER last_known_addr { host: "127.2.110.252" port: 41235 } attrs { promote: true } } }
I20260926 06:15:24.367995 9156 sys_catalog.cc:458] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644 [sys.catalog]: This master's current role is: LEADER
W20260926 06:15:24.368760 8583 consensus_peers.cc:487] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644 -> Peer f5662db075fb4054a7d7380e5834eb1f (127.2.110.252:41235): Couldn't send request to peer f5662db075fb4054a7d7380e5834eb1f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 06:15:24.368901 9206 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f5662db075fb4054a7d7380e5834eb1f (127.2.110.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } peers { permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: NON_VOTER last_known_addr { host: "127.2.110.253" port: 35513 } attrs { promote: true } } peers { permanent_uuid: "f5662db075fb4054a7d7380e5834eb1f" member_type: NON_VOTER last_known_addr { host: "127.2.110.252" port: 41235 } attrs { promote: true } } }
I20260926 06:15:24.369019 9344 mvcc.cc:205] Tried to move back new op lower bound from 7333492011935387648 to 7333492011851415552. Current Snapshot: MvccSnapshot[applied={T|T < 7333492011861999616}]
I20260926 06:15:24.370110 9352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eed8ee75c5394469994882db1a963644" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } peers { permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: NON_VOTER last_known_addr { host: "127.2.110.253" port: 35513 } attrs { promote: true } } peers { permanent_uuid: "f5662db075fb4054a7d7380e5834eb1f" member_type: NON_VOTER last_known_addr { host: "127.2.110.252" port: 41235 } attrs { promote: true } } }
I20260926 06:15:24.370208 9352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088 [sys.catalog]: This master's current role is: LEARNER
I20260926 06:15:24.370621 9235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eed8ee75c5394469994882db1a963644. Latest consensus state: current_term: 1 leader_uuid: "eed8ee75c5394469994882db1a963644" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } peers { permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: NON_VOTER last_known_addr { host: "127.2.110.253" port: 35513 } attrs { promote: true } } peers { permanent_uuid: "f5662db075fb4054a7d7380e5834eb1f" member_type: NON_VOTER last_known_addr { host: "127.2.110.252" port: 41235 } attrs { promote: true } } }
I20260926 06:15:24.370767 9235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088 [sys.catalog]: This master's current role is: LEARNER
I20260926 06:15:24.371209 9250 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260926 06:15:24.371338 9250 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260926 06:15:24.371387 9250 tablet_replica.cc:330] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f: stopping tablet replica
I20260926 06:15:24.371948 9158 raft_consensus.cc:1050] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644: attempting to promote NON_VOTER 4a5a9e72e816463b9cfcb45b2f858088 to VOTER
I20260926 06:15:24.372265 9158 consensus_queue.cc:237] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } peers { permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35513 } attrs { promote: false } } peers { permanent_uuid: "f5662db075fb4054a7d7380e5834eb1f" member_type: NON_VOTER last_known_addr { host: "127.2.110.252" port: 41235 } attrs { promote: true } }
W20260926 06:15:24.373137 8583 consensus_peers.cc:487] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644 -> Peer f5662db075fb4054a7d7380e5834eb1f (127.2.110.252:41235): Couldn't send request to peer f5662db075fb4054a7d7380e5834eb1f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 06:15:24.373771 9206 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088 [term 1 LEARNER]: Refusing update from remote peer eed8ee75c5394469994882db1a963644: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 06:15:24.374017 9156 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35513 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260926 06:15:24.374843 9156 raft_consensus.cc:2932] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 4a5a9e72e816463b9cfcb45b2f858088 (127.2.110.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } peers { permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35513 } attrs { promote: false } } peers { permanent_uuid: "f5662db075fb4054a7d7380e5834eb1f" member_type: NON_VOTER last_known_addr { host: "127.2.110.252" port: 41235 } attrs { promote: true } } }
I20260926 06:15:24.376062 9156 sys_catalog.cc:455] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "eed8ee75c5394469994882db1a963644" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } peers { permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35513 } attrs { promote: false } } peers { permanent_uuid: "f5662db075fb4054a7d7380e5834eb1f" member_type: NON_VOTER last_known_addr { host: "127.2.110.252" port: 41235 } attrs { promote: true } } }
I20260926 06:15:24.376250 9156 sys_catalog.cc:458] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644 [sys.catalog]: This master's current role is: LEADER
I20260926 06:15:24.376348 9156 sys_catalog.cc:455] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eed8ee75c5394469994882db1a963644" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } peers { permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35513 } attrs { promote: false } } peers { permanent_uuid: "f5662db075fb4054a7d7380e5834eb1f" member_type: NON_VOTER last_known_addr { host: "127.2.110.252" port: 41235 } attrs { promote: true } } }
I20260926 06:15:24.376398 9156 sys_catalog.cc:458] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644 [sys.catalog]: This master's current role is: LEADER
I20260926 06:15:24.376870 9206 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 4a5a9e72e816463b9cfcb45b2f858088 (127.2.110.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } peers { permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35513 } attrs { promote: false } } peers { permanent_uuid: "f5662db075fb4054a7d7380e5834eb1f" member_type: NON_VOTER last_known_addr { host: "127.2.110.252" port: 41235 } attrs { promote: true } } }
I20260926 06:15:24.377945 9235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eed8ee75c5394469994882db1a963644" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } peers { permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35513 } attrs { promote: false } } peers { permanent_uuid: "f5662db075fb4054a7d7380e5834eb1f" member_type: NON_VOTER last_known_addr { host: "127.2.110.252" port: 41235 } attrs { promote: true } } }
I20260926 06:15:24.378038 9235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 06:15:24.386708 9250 master.cc:552] Master@127.2.110.252:41235 shutdown complete.
I20260926 06:15:24.386778 9250 master_runner.cc:305] Clearing existing system tablet
I20260926 06:15:24.386929 9250 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:24.386989 9250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:24.387931 9250 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:15:24.388634 9354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:24.388940 9250 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:24.389096 9250 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:24.389147 9250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-2/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-2/wal
uuid: "f5662db075fb4054a7d7380e5834eb1f"
format_stamp: "Formatted at 2026-09-26 06:15:24 on dist-test-slave-hndd"
server_key: "becafeb546e706c72ac63128f480ad70"
server_key_iv: "e8b9a305279b1c7293af2b6d9650ce5a"
server_key_version: "encryptionkey@0"
I20260926 06:15:24.390412 9250 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:15:24.392010 9250 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 06:15:24.392084 9250 log.cc:1196] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 06:15:24.392320 9250 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f: Deleting consensus metadata
I20260926 06:15:24.392454 9250 master_runner.cc:315] Copying system tablet from 127.2.110.254:46749
I20260926 06:15:24.393081 9250 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f: tablet copy: Beginning tablet copy session from remote peer at address 127.2.110.254:46749
I20260926 06:15:24.400665 8638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.393515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49346 (local address 127.2.110.254:46749)
0926 06:15:24.393566 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:24.393568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:24.393683 (+ 115us) server_negotiation.cc:407] Connection header received
0926 06:15:24.393737 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:24.393738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:24.393751 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:24.393832 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:24.394194 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.394975 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.396017 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.396307 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.396380 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.397596 (+ 1216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:24.397597 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:24.397597 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:24.397605 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:24.399426 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:24.399836 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:24.399837 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:24.399838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:24.399909 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:24.400119 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:24.400120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:24.400120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:24.400255 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:24.400323 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:24.400495 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 06:15:24.400574 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 06:15:24.400682 9360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.393472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:46749 (local address 127.0.0.1:49346)
0926 06:15:24.393602 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:24.393613 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:24.393680 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:24.393853 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:24.393854 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:24.394061 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:24.394111 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:24.394113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.395072 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.395073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:24.395925 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:24.395927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.396393 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.396394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:24.396467 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.397020 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:24.397025 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:24.397506 (+ 481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:24.399528 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:24.399530 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:24.399531 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:24.399756 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:24.399984 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:24.399985 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:24.399985 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:24.400049 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:24.400325 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:24.400326 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:24.400380 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 06:15:24.400455 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 06:15:24.400564 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":75,"threads_started":1}
I20260926 06:15:24.401494 8618 tablet_copy_service.cc:140] P eed8ee75c5394469994882db1a963644: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f5662db075fb4054a7d7380e5834eb1f ({username='oryx', principal='oryx/127.2.110.252@KRBTEST.COM'} at 127.0.0.1:49346)
I20260926 06:15:24.401561 8618 tablet_copy_service.cc:161] P eed8ee75c5394469994882db1a963644: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f5662db075fb4054a7d7380e5834eb1f at {username='oryx', principal='oryx/127.2.110.252@KRBTEST.COM'} at 127.0.0.1:49346: session id = f5662db075fb4054a7d7380e5834eb1f-00000000000000000000000000000000
I20260926 06:15:24.402158 8618 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 06:15:24.402647 9250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 06:15:24.403939 9250 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f: tablet copy: Starting download of 0 data blocks...
I20260926 06:15:24.404086 9250 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f: tablet copy: Starting download of 1 WAL segments...
I20260926 06:15:24.404964 9250 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 06:15:24.406076 8618 tablet_copy_service.cc:342] P eed8ee75c5394469994882db1a963644: Request end of tablet copy session f5662db075fb4054a7d7380e5834eb1f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.110.252@KRBTEST.COM'} at 127.0.0.1:49346
I20260926 06:15:24.406142 8618 tablet_copy_service.cc:433] P eed8ee75c5394469994882db1a963644: ending tablet copy session f5662db075fb4054a7d7380e5834eb1f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f5662db075fb4054a7d7380e5834eb1f
I20260926 06:15:24.409087 9250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:24.410228 9361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:24.410252 9364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:24.410580 9250 server_base.cc:1034] running on GCE node
W20260926 06:15:24.413514 9362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:24.413717 9250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:24.413888 9250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:24.415059 9250 hybrid_clock.cc:648] HybridClock initialized: now 1790403324415042 us; error 38 us; skew 500 ppm
Sep 26 06:15:24 dist-test-slave-hndd krb5kdc[8553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403324, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.110.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:24.416759 9250 init.cc:377] Logged in from keytab as oryx/127.2.110.252@KRBTEST.COM (short username oryx)
I20260926 06:15:24.417050 9250 webserver.cc:460] Webserver started at http://127.2.110.252:35891/ using document root <none> and password file <none>
I20260926 06:15:24.417177 9250 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:24.417217 9250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:24.418059 9250 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260926 06:15:24.418674 9371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:24.418788 9250 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 06:15:24.418826 9250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-2/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-2/wal
uuid: "f5662db075fb4054a7d7380e5834eb1f"
format_stamp: "Formatted at 2026-09-26 06:15:24 on dist-test-slave-hndd"
server_key: "becafeb546e706c72ac63128f480ad70"
server_key_iv: "e8b9a305279b1c7293af2b6d9650ce5a"
server_key_version: "encryptionkey@0"
I20260926 06:15:24.418890 9250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:24.429085 9250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:24.431082 9250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:24.434163 9250 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.252:41235
I20260926 06:15:24.434289 9423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.252:41235 every 8 connection(s)
I20260926 06:15:24.434502 9250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/master-2/data/info.pb
I20260926 06:15:24.438388 9424 sys_catalog.cc:263] Verifying existing consensus state
I20260926 06:15:24.438931 9424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f: Bootstrap starting.
I20260926 06:15:24.441390 9424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 06:15:24.441634 9424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f: Bootstrap complete.
I20260926 06:15:24.441830 9424 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } peers { permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35513 } attrs { promote: false } } peers { permanent_uuid: "f5662db075fb4054a7d7380e5834eb1f" member_type: NON_VOTER last_known_addr { host: "127.2.110.252" port: 41235 } attrs { promote: true } }
I20260926 06:15:24.441926 9424 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f5662db075fb4054a7d7380e5834eb1f, State: Initialized, Role: LEARNER
I20260926 06:15:24.441967 9424 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } peers { permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35513 } attrs { promote: false } } peers { permanent_uuid: "f5662db075fb4054a7d7380e5834eb1f" member_type: NON_VOTER last_known_addr { host: "127.2.110.252" port: 41235 } attrs { promote: true } }
I20260926 06:15:24.442154 9424 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:15:24.442168 9427 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } peers { permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35513 } attrs { promote: false } } peers { permanent_uuid: "f5662db075fb4054a7d7380e5834eb1f" member_type: NON_VOTER last_known_addr { host: "127.2.110.252" port: 41235 } attrs { promote: true } } }
I20260926 06:15:24.442420 9427 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f [sys.catalog]: This master's current role is: LEARNER
I20260926 06:15:24.443912 9438 catalog_manager.cc:1259] Loaded cluster ID: b50e97be6eac4bccbae81dfbad9212a1
I20260926 06:15:24.443974 9438 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f: loading cluster ID for follower catalog manager: success
I20260926 06:15:24.445191 9438 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f: acquiring CA information for follower catalog manager: success
I20260926 06:15:24.445515 9438 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 06:15:24.457935 9343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:24.458194 9343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:24.458246 9343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:24.459555 9343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:24.459625 9343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:24.459683 9343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.193
I20260926 06:15:24.461236 9343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:41555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/oryx_127.2.110.193.keytab
--principal=oryx/127.2.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.193:40801
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.110.193
--webserver_port=40447
--tserver_master_addrs=127.2.110.254:46749,127.2.110.253:35513,127.2.110.252:41235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.193
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:24.461529 9343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:24.461772 9343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:24.467139 9443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:24.467288 9343 server_base.cc:1034] running on GCE node
W20260926 06:15:24.467126 9444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:24.467130 9446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:24.467656 9343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:24.468320 9343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:24.469502 9343 hybrid_clock.cc:648] HybridClock initialized: now 1790403324469478 us; error 42 us; skew 500 ppm
Sep 26 06:15:24 dist-test-slave-hndd krb5kdc[8553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403324, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:24.472616 9343 init.cc:377] Logged in from keytab as oryx/127.2.110.193@KRBTEST.COM (short username oryx)
I20260926 06:15:24.473100 9343 webserver.cc:460] Webserver started at http://127.2.110.193:40447/ using document root <none> and password file <none>
I20260926 06:15:24.473412 9343 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:24.473502 9343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:24.475034 9343 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:24.475826 9453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:24.476255 9343 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:24.476400 9343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-0/wal
uuid: "8217ae7c8e30427aa59ee96d67f4bf91"
format_stamp: "Formatted at 2026-09-26 06:15:23 on dist-test-slave-hndd"
server_key: "51fb783c79dd99c5f68b58c717542e01"
server_key_iv: "ec67182dac02997f2140a9cbf153bfec"
server_key_version: "encryptionkey@0"
I20260926 06:15:24.476787 9343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:24.500365 9343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:24.502892 9343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:24.503167 9343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:24.503436 9343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:15:24.503783 9343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:15:24.503850 9343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:24.503896 9343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:15:24.503926 9343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:24.511704 9343 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.193:40801
I20260926 06:15:24.511775 9566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.193:40801 every 8 connection(s)
I20260926 06:15:24.512172 9343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
I20260926 06:15:24.514757 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 9343
I20260926 06:15:24.514935 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 8794
Sep 26 06:15:24 dist-test-slave-hndd krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403324, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.110.193@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM
I20260926 06:15:24.520509 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.194:45563
--local_ip_for_outbound_sockets=127.2.110.194
--tserver_master_addrs=127.2.110.254:46749
--webserver_port=43489
--webserver_interface=127.2.110.194
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/oryx_127.2.110.194.keytab
--principal=oryx/127.2.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:41555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.110.254:46749,127.2.110.253:35513
--tserver_master_addrs=127.2.110.254:46749,127.2.110.253:35513,127.2.110.252:41235 with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 06:15:24.523797 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.513313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:46749 (local address 127.2.110.193:34499)
0926 06:15:24.513469 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:24.513478 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:24.514456 (+ 978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:24.514959 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:24.514960 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:24.515236 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:24.515505 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:24.515510 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.516287 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.516288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:24.517276 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:24.517278 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.517747 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.517749 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:24.517880 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.518464 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:24.518473 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:24.520287 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:24.522139 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:24.522147 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:24.522150 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:24.522470 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:24.522679 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:24.522680 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:24.522680 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:24.522798 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:24.523096 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:24.523103 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:24.523265 (+ 162us) client_negotiation.cc:769] Sending connection context
0926 06:15:24.523379 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 06:15:24.523554 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":84,"threads_started":1}
I20260926 06:15:24.524271 8638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.513476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:34499 (local address 127.2.110.254:46749)
0926 06:15:24.513541 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:24.513543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:24.513553 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:15:24.514691 (+ 1138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:24.514693 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:24.514712 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:24.514783 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:24.515606 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.516189 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.517387 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.517645 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.517736 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.520401 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:24.520403 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:24.520404 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:24.520411 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:24.522041 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:24.522558 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:24.522559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:24.522559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:24.522618 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:24.522872 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:24.522873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:24.522873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:24.523010 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:24.523073 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:24.524093 (+ 1020us) server_negotiation.cc:299] Negotiation successful
0926 06:15:24.524182 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 06:15:24.525017 9568 heartbeater.cc:347] Connected to a master server at 127.2.110.254:46749
I20260926 06:15:24.525125 9568 heartbeater.cc:464] Registering TS with master...
I20260926 06:15:24.525377 9568 heartbeater.cc:511] Master 127.2.110.254:46749 requested a full tablet report, sending...
I20260926 06:15:24.525993 8598 ts_manager.cc:194] Re-registered known tserver with Master: 8217ae7c8e30427aa59ee96d67f4bf91 (127.2.110.193:40801)
Sep 26 06:15:24 dist-test-slave-hndd krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403324, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.110.193@KRBTEST.COM for oryx/127.2.110.252@KRBTEST.COM
I20260926 06:15:24.526574 8598 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.110.193@KRBTEST.COM'} at 127.2.110.193:34499
I20260926 06:15:24.529094 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.513088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:42897 (local address 127.2.110.252:41235)
0926 06:15:24.513702 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:24.513704 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:24.513714 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:15:24.514662 (+ 948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:24.514664 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:24.514679 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:24.514757 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:24.521892 (+ 7135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.522920 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.524077 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.524389 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.524462 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.526408 (+ 1946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:24.526410 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:24.526411 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:24.526419 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:24.527905 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:24.528284 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:24.528285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:24.528286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:24.528351 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:24.528555 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:24.528556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:24.528556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:24.528673 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:24.528751 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:24.528918 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 06:15:24.529001 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":64,"threads_started":1}
I20260926 06:15:24.529107 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.512984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.252:41235 (local address 127.2.110.193:42897)
0926 06:15:24.513468 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:24.513478 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:24.514476 (+ 998us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:24.514856 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:24.514861 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:24.521694 (+ 6833us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:24.521782 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:24.521785 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.523038 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.523039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:24.523987 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:24.523991 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.524461 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.524461 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:24.524531 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.524938 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:24.524943 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:24.526293 (+ 1350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:24.528013 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:24.528015 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:24.528015 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:24.528203 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:24.528426 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:24.528426 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:24.528427 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:24.528488 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:24.528742 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:24.528742 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:24.528785 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 06:15:24.528877 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 06:15:24.528990 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"mutex_wait_us":4,"thread_start_us":120,"threads_started":1}
I20260926 06:15:24.529491 9567 heartbeater.cc:347] Connected to a master server at 127.2.110.252:41235
I20260926 06:15:24.529547 9567 heartbeater.cc:464] Registering TS with master...
I20260926 06:15:24.529623 9567 heartbeater.cc:511] Master 127.2.110.252:41235 requested a full tablet report, sending...
I20260926 06:15:24.529994 9388 ts_manager.cc:194] Registered new tserver with Master: 8217ae7c8e30427aa59ee96d67f4bf91 (127.2.110.193:40801)
Sep 26 06:15:24 dist-test-slave-hndd krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403324, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.110.193@KRBTEST.COM for oryx/127.2.110.253@KRBTEST.COM
I20260926 06:15:24.538022 9247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.513057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:54613 (local address 127.2.110.253:35513)
0926 06:15:24.513113 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:24.513115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:24.513324 (+ 209us) server_negotiation.cc:407] Connection header received
0926 06:15:24.514840 (+ 1516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:24.514842 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:24.514852 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:24.514914 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:24.520672 (+ 5758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.521488 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.522935 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.523252 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.523296 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.534943 (+ 11647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:24.534946 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:24.534947 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:24.534955 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:24.536808 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:24.537195 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:24.537200 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:24.537201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:24.537249 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:24.537483 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:24.537484 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:24.537488 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:24.537604 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:24.537682 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:24.537842 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 06:15:24.537924 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 06:15:24.538342 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.512982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.253:35513 (local address 127.2.110.193:54613)
0926 06:15:24.513185 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:24.513231 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:24.514456 (+ 1225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:24.520287 (+ 5831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:24.520289 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:24.520479 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:24.520553 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:24.520555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.521906 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.521907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:24.522813 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:24.522816 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.523381 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.523382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:24.523453 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.533517 (+ 10064us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:24.533521 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:24.534834 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:24.536928 (+ 2094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:24.536929 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:24.536930 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:24.537109 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:24.537319 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:24.537320 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:24.537321 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:24.537409 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:24.537673 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:24.537674 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:24.537722 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 06:15:24.537806 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 06:15:24.537932 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":115,"threads_started":1}
I20260926 06:15:24.538834 9569 heartbeater.cc:347] Connected to a master server at 127.2.110.253:35513
I20260926 06:15:24.538900 9569 heartbeater.cc:464] Registering TS with master...
I20260926 06:15:24.538968 9569 heartbeater.cc:511] Master 127.2.110.253:35513 requested a full tablet report, sending...
I20260926 06:15:24.539357 9196 ts_manager.cc:194] Registered new tserver with Master: 8217ae7c8e30427aa59ee96d67f4bf91 (127.2.110.193:40801)
W20260926 06:15:24.615971 9575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:24.616269 9575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:24.616302 9575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:24.617734 9575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:24.617781 9575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:24.617810 9575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.194
I20260926 06:15:24.619374 9575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:41555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/oryx_127.2.110.194.keytab
--principal=oryx/127.2.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.194:45563
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.110.194
--webserver_port=43489
--tserver_master_addrs=127.2.110.254:46749,127.2.110.253:35513,127.2.110.252:41235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.194
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:24.619575 9575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:24.619771 9575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:24.624963 9583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:24.625084 9575 server_base.cc:1034] running on GCE node
W20260926 06:15:24.624969 9580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:24.624969 9581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:24.625397 9575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:24.625622 9575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:24.626761 9575 hybrid_clock.cc:648] HybridClock initialized: now 1790403324626753 us; error 32 us; skew 500 ppm
Sep 26 06:15:24 dist-test-slave-hndd krb5kdc[8553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403324, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:24.629750 9575 init.cc:377] Logged in from keytab as oryx/127.2.110.194@KRBTEST.COM (short username oryx)
I20260926 06:15:24.630385 9575 webserver.cc:460] Webserver started at http://127.2.110.194:43489/ using document root <none> and password file <none>
I20260926 06:15:24.630641 9575 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:24.630730 9575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:24.632635 9575 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:15:24.633461 9590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:24.633634 9575 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 06:15:24.633733 9575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-1/wal
uuid: "2f74a0910ea94f6aa814996cc6eff5be"
format_stamp: "Formatted at 2026-09-26 06:15:23 on dist-test-slave-hndd"
server_key: "87811499738e6a86ac561019b5fd7416"
server_key_iv: "c7dc60da6e563d1fec8491c67601451f"
server_key_version: "encryptionkey@0"
I20260926 06:15:24.634150 9575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:24.657722 9575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:24.659953 9575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:24.660136 9575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:24.660398 9575 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:15:24.660717 9575 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:15:24.660775 9575 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:24.660820 9575 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:15:24.660844 9575 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:24.667500 9575 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.194:45563
I20260926 06:15:24.667552 9703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.194:45563 every 8 connection(s)
I20260926 06:15:24.667917 9575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
I20260926 06:15:24.670095 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 9575
I20260926 06:15:24.670251 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 8928
I20260926 06:15:24.674762 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.195:38407
--local_ip_for_outbound_sockets=127.2.110.195
--tserver_master_addrs=127.2.110.254:46749
--webserver_port=33753
--webserver_interface=127.2.110.195
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/oryx_127.2.110.195.keytab
--principal=oryx/127.2.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:41555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.110.254:46749,127.2.110.253:35513
--tserver_master_addrs=127.2.110.254:46749,127.2.110.253:35513,127.2.110.252:41235 with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 06:15:24 dist-test-slave-hndd krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403324, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.110.194@KRBTEST.COM for oryx/127.2.110.252@KRBTEST.COM
Sep 26 06:15:24 dist-test-slave-hndd krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403324, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.110.194@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM
Sep 26 06:15:24 dist-test-slave-hndd krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403324, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.110.194@KRBTEST.COM for oryx/127.2.110.253@KRBTEST.COM
I20260926 06:15:24.684590 9709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.668704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:46749 (local address 127.2.110.194:36093)
0926 06:15:24.668890 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:24.668932 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:24.669606 (+ 674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:24.669899 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:24.669901 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:24.674072 (+ 4171us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:24.674442 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:24.674451 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.675522 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.675523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:24.676870 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:24.676873 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.677320 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.677321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:24.677441 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.678163 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:24.678172 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:24.680408 (+ 2236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:24.682705 (+ 2297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:24.682706 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:24.682707 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:24.682984 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:24.683234 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:24.683235 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:24.683236 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:24.683408 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:24.683738 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:24.683744 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:24.683938 (+ 194us) client_negotiation.cc:769] Sending connection context
0926 06:15:24.684058 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 06:15:24.684266 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":104,"thread_start_us":100,"threads_started":1}
I20260926 06:15:24.684725 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.669203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:42881 (local address 127.2.110.252:41235)
0926 06:15:24.669254 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:24.669256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:24.669265 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:15:24.669706 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:24.669708 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:24.669720 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:24.669774 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:24.674579 (+ 4805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.675339 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.676825 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.677152 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.677226 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.680059 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:24.680061 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:24.680061 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:24.680070 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:24.682349 (+ 2279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:24.683842 (+ 1493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:24.683844 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:24.683844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:24.683901 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:24.684104 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:24.684105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:24.684105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:24.684244 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:24.684314 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:24.684502 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 06:15:24.684603 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 06:15:24.684953 9247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.668687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:34905 (local address 127.2.110.253:35513)
0926 06:15:24.668758 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:24.668760 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:24.669029 (+ 269us) server_negotiation.cc:407] Connection header received
0926 06:15:24.669706 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:24.669708 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:24.669720 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:24.669752 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:24.674571 (+ 4819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.675360 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.677409 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.677704 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.677777 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.681321 (+ 3544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:24.681323 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:24.681323 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:24.681357 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:24.683747 (+ 2390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:24.684109 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:24.684110 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:24.684110 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:24.684160 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:24.684359 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:24.684360 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:24.684360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:24.684494 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:24.684535 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:24.684723 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 06:15:24.684831 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 06:15:24.685149 8638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.668732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:36093 (local address 127.2.110.254:46749)
0926 06:15:24.668819 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:24.668821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:24.669031 (+ 210us) server_negotiation.cc:407] Connection header received
0926 06:15:24.669746 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:24.669747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:24.669759 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:24.669790 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:24.674571 (+ 4781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.675366 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.676967 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.677255 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.677320 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.680515 (+ 3195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:24.680517 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:24.680518 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:24.680527 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:24.682619 (+ 2092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:24.683089 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:24.683090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:24.683090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:24.683159 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:24.683496 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:24.683497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:24.683497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:24.683662 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:24.683736 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:24.684868 (+ 1132us) server_negotiation.cc:299] Negotiation successful
0926 06:15:24.684963 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 06:15:24.685252 9708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.668704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.253:35513 (local address 127.2.110.194:34905)
0926 06:15:24.668896 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:24.668932 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:24.669583 (+ 651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:24.669860 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:24.669865 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:24.674072 (+ 4207us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:24.674472 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:24.674474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.675522 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.675523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:24.676870 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:24.676873 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.677792 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.677792 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:24.677866 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.679704 (+ 1838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:24.679709 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:24.681216 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:24.683809 (+ 2593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:24.683810 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:24.683810 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:24.684022 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:24.684231 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:24.684232 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:24.684232 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:24.684285 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:24.684582 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:24.684583 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:24.684617 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 06:15:24.685042 (+ 425us) client_negotiation.cc:240] Negotiation successful
0926 06:15:24.685153 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":317,"thread_start_us":81,"threads_started":1}
I20260926 06:15:24.685720 9705 heartbeater.cc:347] Connected to a master server at 127.2.110.254:46749
I20260926 06:15:24.685699 9706 heartbeater.cc:347] Connected to a master server at 127.2.110.253:35513
I20260926 06:15:24.685818 9710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.669018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.252:41235 (local address 127.2.110.194:42881)
0926 06:15:24.669134 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:24.669143 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:24.669585 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:24.671918 (+ 2333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:24.671920 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:24.674181 (+ 2261us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:24.674441 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:24.674449 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.675710 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.675711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:24.676712 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:24.676715 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.677252 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.677253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:24.677441 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.678244 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:24.678249 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:24.679967 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:24.682455 (+ 2488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:24.682465 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:24.682469 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:24.682881 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:24.683975 (+ 1094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:24.683975 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:24.683976 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:24.684031 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:24.684347 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:24.684348 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:24.684389 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 06:15:24.685397 (+ 1008us) client_negotiation.cc:240] Negotiation successful
0926 06:15:24.685503 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":35,"thread_start_us":60,"threads_started":1}
I20260926 06:15:24.685860 9706 heartbeater.cc:464] Registering TS with master...
I20260926 06:15:24.685860 9705 heartbeater.cc:464] Registering TS with master...
I20260926 06:15:24.686079 9705 heartbeater.cc:511] Master 127.2.110.254:46749 requested a full tablet report, sending...
I20260926 06:15:24.686077 9706 heartbeater.cc:511] Master 127.2.110.253:35513 requested a full tablet report, sending...
I20260926 06:15:24.686510 8598 ts_manager.cc:194] Re-registered known tserver with Master: 2f74a0910ea94f6aa814996cc6eff5be (127.2.110.194:45563)
I20260926 06:15:24.686535 9196 ts_manager.cc:194] Registered new tserver with Master: 2f74a0910ea94f6aa814996cc6eff5be (127.2.110.194:45563)
I20260926 06:15:24.686619 9704 heartbeater.cc:347] Connected to a master server at 127.2.110.252:41235
I20260926 06:15:24.686667 9704 heartbeater.cc:464] Registering TS with master...
I20260926 06:15:24.686726 9704 heartbeater.cc:511] Master 127.2.110.252:41235 requested a full tablet report, sending...
I20260926 06:15:24.686975 9388 ts_manager.cc:194] Registered new tserver with Master: 2f74a0910ea94f6aa814996cc6eff5be (127.2.110.194:45563)
I20260926 06:15:24.687017 8598 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.110.194@KRBTEST.COM'} at 127.2.110.194:36093
W20260926 06:15:24.766278 9711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:24.766531 9711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:24.766577 9711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:24.767791 9711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:24.767866 9711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:24.767915 9711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.195
I20260926 06:15:24.769475 9711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:41555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/krb5kdc/oryx_127.2.110.195.keytab
--principal=oryx/127.2.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.195:38407
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.110.195
--webserver_port=33753
--tserver_master_addrs=127.2.110.254:46749,127.2.110.253:35513,127.2.110.252:41235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.195
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:24.769757 9711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:24.770002 9711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:24.775282 9719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:24.775368 9711 server_base.cc:1034] running on GCE node
W20260926 06:15:24.775316 9716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:24.775315 9717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:24.775792 9711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:24.776012 9711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:24.777166 9711 hybrid_clock.cc:648] HybridClock initialized: now 1790403324777141 us; error 41 us; skew 500 ppm
Sep 26 06:15:24 dist-test-slave-hndd krb5kdc[8553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403324, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:24.780022 9711 init.cc:377] Logged in from keytab as oryx/127.2.110.195@KRBTEST.COM (short username oryx)
I20260926 06:15:24.780562 9711 webserver.cc:460] Webserver started at http://127.2.110.195:33753/ using document root <none> and password file <none>
I20260926 06:15:24.780848 9711 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:24.780933 9711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:24.782589 9711 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:24.783289 9726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:24.783509 9711 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:24.783607 9711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-2/wal
uuid: "96a7adaf68cd451ba1388e6550cf780c"
format_stamp: "Formatted at 2026-09-26 06:15:23 on dist-test-slave-hndd"
server_key: "3d2578838376e92c2d3393695df2c5b8"
server_key_iv: "808b459b1670427d4f45f3ac7b790217"
server_key_version: "encryptionkey@0"
I20260926 06:15:24.784153 9711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:24.792459 9711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:24.794713 9711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:24.794836 9711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:24.795043 9711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:15:24.795373 9711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:15:24.795411 9711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:24.795460 9711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:15:24.795486 9711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:24.801314 9711 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.195:38407
I20260926 06:15:24.801394 9839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.195:38407 every 8 connection(s)
I20260926 06:15:24.801801 9711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
Sep 26 06:15:24 dist-test-slave-hndd krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403324, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.110.195@KRBTEST.COM for oryx/127.2.110.253@KRBTEST.COM
Sep 26 06:15:24 dist-test-slave-hndd krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403324, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.110.195@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM
Sep 26 06:15:24 dist-test-slave-hndd krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403324, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.110.195@KRBTEST.COM for oryx/127.2.110.252@KRBTEST.COM
I20260926 06:15:24.811965 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 9711
I20260926 06:15:24.816783 8638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.802607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:41389 (local address 127.2.110.254:46749)
0926 06:15:24.802674 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:24.802677 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:24.802876 (+ 199us) server_negotiation.cc:407] Connection header received
0926 06:15:24.803662 (+ 786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:24.803663 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:24.803675 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:24.803741 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:24.805416 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.805980 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.807116 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.807432 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.807473 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.812052 (+ 4579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:24.812056 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:24.812056 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:24.812068 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:24.814766 (+ 2698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:24.815841 (+ 1075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:24.815842 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:24.815843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:24.815907 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:24.816114 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:24.816115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:24.816115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:24.816270 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:24.816316 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:24.816587 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 06:15:24.816674 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 06:15:24.816825 9247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.802833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:33139 (local address 127.2.110.253:35513)
0926 06:15:24.802881 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:24.802883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:24.802893 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:15:24.803535 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:24.803536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:24.803547 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:24.803601 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:24.805033 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.805607 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.807035 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.807376 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.807421 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.812511 (+ 5090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:24.812513 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:24.812514 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:24.812524 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:24.814798 (+ 2274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:24.815502 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:24.815504 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:24.815505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:24.815567 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:24.815941 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:24.815942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:24.815942 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:24.816103 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:24.816149 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:24.816596 (+ 447us) server_negotiation.cc:299] Negotiation successful
0926 06:15:24.816688 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 06:15:24.817595 9844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.802586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.253:35513 (local address 127.2.110.195:33139)
0926 06:15:24.802749 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:24.802792 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:24.803425 (+ 633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:24.804429 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:24.804430 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:24.804702 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:24.804913 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:24.804918 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.805711 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.805712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:24.806909 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:24.806912 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.807526 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.807527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:24.807698 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.808413 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:24.808424 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:24.812315 (+ 3891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:24.814910 (+ 2595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:24.814958 (+ 48us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:24.814963 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:24.815400 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:24.815683 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:24.815684 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:24.815684 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:24.815847 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:24.816211 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:24.816218 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:24.816420 (+ 202us) client_negotiation.cc:769] Sending connection context
0926 06:15:24.816925 (+ 505us) client_negotiation.cc:240] Negotiation successful
0926 06:15:24.817196 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"spinlock_wait_cycles":37248,"thread_start_us":115,"threads_started":1}
I20260926 06:15:24.817597 9845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.802586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:46749 (local address 127.2.110.195:41389)
0926 06:15:24.802756 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:24.802792 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:24.803425 (+ 633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:24.803798 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:24.803802 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:24.804713 (+ 911us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:24.804914 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:24.804919 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.806074 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.806075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:24.807031 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:24.807032 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.807552 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.807552 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:24.807698 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.808833 (+ 1135us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:24.808837 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:24.811841 (+ 3004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:24.814881 (+ 3040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:24.815178 (+ 297us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:24.815178 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:24.815743 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:24.815986 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:24.815986 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:24.815987 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:24.816045 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:24.816367 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:24.816367 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:24.816420 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 06:15:24.816929 (+ 509us) client_negotiation.cc:240] Negotiation successful
0926 06:15:24.817296 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":187,"thread_start_us":84,"threads_started":1}
I20260926 06:15:24.818123 9841 heartbeater.cc:347] Connected to a master server at 127.2.110.254:46749
I20260926 06:15:24.818199 9842 heartbeater.cc:347] Connected to a master server at 127.2.110.253:35513
I20260926 06:15:24.818254 9841 heartbeater.cc:464] Registering TS with master...
I20260926 06:15:24.818255 9842 heartbeater.cc:464] Registering TS with master...
I20260926 06:15:24.818495 9841 heartbeater.cc:511] Master 127.2.110.254:46749 requested a full tablet report, sending...
I20260926 06:15:24.818495 9842 heartbeater.cc:511] Master 127.2.110.253:35513 requested a full tablet report, sending...
I20260926 06:15:24.818928 8598 ts_manager.cc:194] Re-registered known tserver with Master: 96a7adaf68cd451ba1388e6550cf780c (127.2.110.195:38407)
I20260926 06:15:24.819002 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.803081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:53725 (local address 127.2.110.252:41235)
0926 06:15:24.803130 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:24.803132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:24.803140 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:15:24.803792 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:24.803794 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:24.803805 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:24.803866 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:24.806105 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.806887 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.807889 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.808177 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.808220 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.814649 (+ 6429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:24.814651 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:24.814651 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:24.814660 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:24.816434 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:24.817374 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:24.817376 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:24.817376 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:24.817436 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:24.817661 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:24.817661 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:24.817662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:24.817801 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:24.817870 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:24.818837 (+ 967us) server_negotiation.cc:299] Negotiation successful
0926 06:15:24.818925 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 06:15:24.819192 9196 ts_manager.cc:194] Registered new tserver with Master: 96a7adaf68cd451ba1388e6550cf780c (127.2.110.195:38407)
I20260926 06:15:24.819459 8598 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.110.195@KRBTEST.COM'} at 127.2.110.195:41389
I20260926 06:15:24.820250 9846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.802878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.252:41235 (local address 127.2.110.195:53725)
0926 06:15:24.803011 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:24.803021 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:24.803425 (+ 404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:24.803913 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:24.803914 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:24.804751 (+ 837us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:24.804914 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:24.804919 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.806992 (+ 2073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.806993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:24.807775 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:24.807777 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.808301 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.808302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:24.808372 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.808806 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:24.808812 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:24.813432 (+ 4620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:24.816514 (+ 3082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:24.816515 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:24.816516 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:24.816756 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:24.817523 (+ 767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:24.817524 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:24.817524 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:24.817579 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:24.818642 (+ 1063us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:24.818642 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:24.818676 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 06:15:24.819679 (+ 1003us) client_negotiation.cc:240] Negotiation successful
0926 06:15:24.819953 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":3,"thread_start_us":82,"threads_started":1}
I20260926 06:15:24.820554 9840 heartbeater.cc:347] Connected to a master server at 127.2.110.252:41235
I20260926 06:15:24.820605 9840 heartbeater.cc:464] Registering TS with master...
I20260926 06:15:24.820677 9840 heartbeater.cc:511] Master 127.2.110.252:41235 requested a full tablet report, sending...
I20260926 06:15:24.820983 9388 ts_manager.cc:194] Registered new tserver with Master: 96a7adaf68cd451ba1388e6550cf780c (127.2.110.195:38407)
I20260926 06:15:24.822667 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.819353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35300 (local address 127.2.110.252:41235)
0926 06:15:24.819391 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:24.819392 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:24.819398 (+ 6us) server_negotiation.cc:407] Connection header received
0926 06:15:24.819417 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:24.819418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:24.819439 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:24.819495 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:15:24.819685 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.820517 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.821688 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.822333 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.822402 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.822468 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:24.822572 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 06:15:24.822606 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 06:15:24.822715 9155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.819188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.252:41235 (local address 127.0.0.1:35300)
0926 06:15:24.819233 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:24.819243 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:24.819277 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:24.819535 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:24.819536 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:24.819538 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:15:24.819583 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:24.819585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.820628 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.820629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:24.821598 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:24.821600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.822414 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.822415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:24.822491 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.822496 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 06:15:24.822566 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 06:15:24.822622 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 06:15:24.822996 9398 raft_consensus.cc:1203] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260926 06:15:24.823149 9427 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader eed8ee75c5394469994882db1a963644. Latest consensus state: current_term: 1 leader_uuid: "eed8ee75c5394469994882db1a963644" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } peers { permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35513 } attrs { promote: false } } peers { permanent_uuid: "f5662db075fb4054a7d7380e5834eb1f" member_type: NON_VOTER last_known_addr { host: "127.2.110.252" port: 41235 } attrs { promote: true } } }
I20260926 06:15:24.823242 9427 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f [sys.catalog]: This master's current role is: LEARNER
I20260926 06:15:24.830037 9851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.813984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49350 (local address 127.2.110.254:46749)
0926 06:15:24.815635 (+ 1651us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:24.815637 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:24.815644 (+ 7us) server_negotiation.cc:407] Connection header received
0926 06:15:24.815681 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:24.815681 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:24.815696 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:24.820848 (+ 5152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:24.822196 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.823338 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.824422 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.824743 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.824818 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.826742 (+ 1924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:24.826743 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:24.826744 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:24.826751 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:24.828672 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:24.829126 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:24.829127 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:24.829127 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:24.829193 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:24.829435 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:24.829436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:24.829436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:24.829569 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:24.829650 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:24.829829 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 06:15:24.829899 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1590,"thread_start_us":91,"threads_started":1}
I20260926 06:15:24.831676 9853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.814412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35288 (local address 127.2.110.252:41235)
0926 06:15:24.814971 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:24.814972 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:24.822131 (+ 7159us) server_negotiation.cc:407] Connection header received
0926 06:15:24.822169 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:24.822169 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:24.822179 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:24.822256 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:24.823429 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.823999 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.824889 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.825071 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.825151 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.826644 (+ 1493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:24.826646 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:24.826646 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:24.826653 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:24.828688 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:24.829240 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:24.829242 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:24.829243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:24.829304 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:24.831102 (+ 1798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:24.831103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:24.831104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:24.831260 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:24.831358 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:24.831469 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 06:15:24.831557 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":47,"threads_started":1}
I20260926 06:15:24.832120 9247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.823215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54262 (local address 127.2.110.253:35513)
0926 06:15:24.823264 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:24.823266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:24.823528 (+ 262us) server_negotiation.cc:407] Connection header received
0926 06:15:24.823593 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:24.823594 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:24.823605 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:24.823662 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:24.824346 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.824883 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.825867 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.826204 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.826250 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.828275 (+ 2025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:24.828277 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:24.828278 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:24.828287 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:24.830490 (+ 2203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:24.830955 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:24.830956 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:24.830957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:24.831017 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:24.831253 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:24.831255 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:24.831255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:24.831587 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:24.831693 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:24.831961 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 06:15:24.832037 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 06:15:24.831760 8598 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49350:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:15:24.834031 8598 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:15:24.834789 9157 sys_catalog.cc:455] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eed8ee75c5394469994882db1a963644" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } peers { permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35513 } attrs { promote: false } } peers { permanent_uuid: "f5662db075fb4054a7d7380e5834eb1f" member_type: NON_VOTER last_known_addr { host: "127.2.110.252" port: 41235 } attrs { promote: true } } }
I20260926 06:15:24.834893 9157 sys_catalog.cc:458] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644 [sys.catalog]: This master's current role is: LEADER
I20260926 06:15:24.835809 9158 raft_consensus.cc:1050] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644: attempting to promote NON_VOTER f5662db075fb4054a7d7380e5834eb1f to VOTER
I20260926 06:15:24.835994 9158 consensus_queue.cc:237] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } peers { permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35513 } attrs { promote: false } } peers { permanent_uuid: "f5662db075fb4054a7d7380e5834eb1f" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 41235 } attrs { promote: false } }
I20260926 06:15:24.836361 9206 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088 [term 1 FOLLOWER]: Refusing update from remote peer eed8ee75c5394469994882db1a963644: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260926 06:15:24.836469 9398 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f [term 1 LEARNER]: Refusing update from remote peer eed8ee75c5394469994882db1a963644: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260926 06:15:24.836544 9157 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35513 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260926 06:15:24.837443 9158 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5662db075fb4054a7d7380e5834eb1f" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 41235 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260926 06:15:24.837636 9857 raft_consensus.cc:2932] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, f5662db075fb4054a7d7380e5834eb1f (127.2.110.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } peers { permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35513 } attrs { promote: false } } peers { permanent_uuid: "f5662db075fb4054a7d7380e5834eb1f" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 41235 } attrs { promote: false } } }
I20260926 06:15:24.837767 9206 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, f5662db075fb4054a7d7380e5834eb1f (127.2.110.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } peers { permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35513 } attrs { promote: false } } peers { permanent_uuid: "f5662db075fb4054a7d7380e5834eb1f" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 41235 } attrs { promote: false } } }
I20260926 06:15:24.838521 9158 sys_catalog.cc:455] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "eed8ee75c5394469994882db1a963644" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } peers { permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35513 } attrs { promote: false } } peers { permanent_uuid: "f5662db075fb4054a7d7380e5834eb1f" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 41235 } attrs { promote: false } } }
I20260926 06:15:24.838613 9158 sys_catalog.cc:458] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644 [sys.catalog]: This master's current role is: LEADER
I20260926 06:15:24.838632 9235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eed8ee75c5394469994882db1a963644" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } peers { permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35513 } attrs { promote: false } } peers { permanent_uuid: "f5662db075fb4054a7d7380e5834eb1f" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 41235 } attrs { promote: false } } }
I20260926 06:15:24.838716 9235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 06:15:24.838521 9857 sys_catalog.cc:455] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eed8ee75c5394469994882db1a963644" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } peers { permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35513 } attrs { promote: false } } peers { permanent_uuid: "f5662db075fb4054a7d7380e5834eb1f" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 41235 } attrs { promote: false } } }
I20260926 06:15:24.838802 9857 sys_catalog.cc:458] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644 [sys.catalog]: This master's current role is: LEADER
I20260926 06:15:24.839144 9857 sys_catalog.cc:455] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eed8ee75c5394469994882db1a963644" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } peers { permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35513 } attrs { promote: false } } peers { permanent_uuid: "f5662db075fb4054a7d7380e5834eb1f" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 41235 } attrs { promote: false } } }
I20260926 06:15:24.839232 9857 sys_catalog.cc:458] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644 [sys.catalog]: This master's current role is: LEADER
I20260926 06:15:24.839416 9398 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, f5662db075fb4054a7d7380e5834eb1f (127.2.110.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } peers { permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35513 } attrs { promote: false } } peers { permanent_uuid: "f5662db075fb4054a7d7380e5834eb1f" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 41235 } attrs { promote: false } } }
I20260926 06:15:24.840544 9427 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eed8ee75c5394469994882db1a963644" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } peers { permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35513 } attrs { promote: false } } peers { permanent_uuid: "f5662db075fb4054a7d7380e5834eb1f" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 41235 } attrs { promote: false } } }
I20260926 06:15:24.840631 9427 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 06:15:24.844254 9155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.839847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.194:45563 (local address 127.0.0.1:37924)
0926 06:15:24.839919 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:24.839935 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:24.839983 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:24.840937 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:24.840938 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:24.840940 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:15:24.840999 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:24.841000 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.842177 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.842178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:24.843209 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:24.843211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.844026 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.844027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:24.844084 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.844088 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:15:24.844132 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 06:15:24.844179 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":3}
I20260926 06:15:24.844336 9868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.840600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37924 (local address 127.2.110.194:45563)
0926 06:15:24.840750 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:24.840755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:24.840766 (+ 11us) server_negotiation.cc:407] Connection header received
0926 06:15:24.840807 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:24.840809 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:24.840826 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:24.841080 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:15:24.841144 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.842096 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.843296 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.843934 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.844029 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.844176 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:24.844203 (+ 27us) server_negotiation.cc:299] Negotiation successful
0926 06:15:24.844245 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":84,"threads_started":1}
I20260926 06:15:24.844521 9864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.839956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.195:38407 (local address 127.0.0.1:38414)
0926 06:15:24.840165 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:24.840175 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:24.840219 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:24.840928 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:24.840929 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:24.840932 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:15:24.841003 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:24.841005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.842310 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.842311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:24.843420 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:24.843422 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.844296 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.844297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:24.844375 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.844380 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 06:15:24.844414 (+ 34us) client_negotiation.cc:240] Negotiation successful
0926 06:15:24.844454 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":2,"thread_start_us":107,"threads_started":1}
I20260926 06:15:24.844902 9865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.840064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38414 (local address 127.2.110.195:38407)
0926 06:15:24.840252 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:24.840258 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:24.840270 (+ 12us) server_negotiation.cc:407] Connection header received
0926 06:15:24.840319 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:24.840321 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:24.840338 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:24.841079 (+ 741us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:15:24.841136 (+ 57us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.842216 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.843512 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.844200 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.844693 (+ 493us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.844757 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:24.844781 (+ 24us) server_negotiation.cc:299] Negotiation successful
0926 06:15:24.844822 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":80,"threads_started":1}
I20260926 06:15:24.845068 9638 tablet_service.cc:1467] Processing CreateTablet for tablet 55b0ee0531e14410aa1fce3360b4e715 (DEFAULT_TABLE table=test-table [id=f1caf4f22ad54123a47309d9ff52be34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:15:24.845548 9638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55b0ee0531e14410aa1fce3360b4e715. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:24.845554 9774 tablet_service.cc:1467] Processing CreateTablet for tablet 55b0ee0531e14410aa1fce3360b4e715 (DEFAULT_TABLE table=test-table [id=f1caf4f22ad54123a47309d9ff52be34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:15:24.845875 9774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55b0ee0531e14410aa1fce3360b4e715. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:24.845917 9867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.840345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41510 (local address 127.2.110.193:40801)
0926 06:15:24.840985 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:24.840989 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:24.841538 (+ 549us) server_negotiation.cc:407] Connection header received
0926 06:15:24.841652 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:24.841654 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:24.841668 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:24.841712 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:15:24.841915 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.843092 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.844620 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.845439 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.845537 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.845607 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:24.845803 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 06:15:24.845854 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":104,"threads_started":1}
I20260926 06:15:24.846168 9866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.840151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.193:40801 (local address 127.0.0.1:41510)
0926 06:15:24.841446 (+ 1295us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:24.841457 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:24.841567 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:24.841790 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:24.841791 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:24.841793 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:15:24.841830 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:24.841832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.843180 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.843181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:24.844265 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:24.844268 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.845539 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.845540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:24.845672 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.845676 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:15:24.846053 (+ 377us) client_negotiation.cc:240] Negotiation successful
0926 06:15:24.846098 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1276,"thread_start_us":58,"threads_started":1}
I20260926 06:15:24.846772 9501 tablet_service.cc:1467] Processing CreateTablet for tablet 55b0ee0531e14410aa1fce3360b4e715 (DEFAULT_TABLE table=test-table [id=f1caf4f22ad54123a47309d9ff52be34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:15:24.847074 9501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55b0ee0531e14410aa1fce3360b4e715. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:24.850235 9869 tablet_bootstrap.cc:492] T 55b0ee0531e14410aa1fce3360b4e715 P 8217ae7c8e30427aa59ee96d67f4bf91: Bootstrap starting.
I20260926 06:15:24.850562 9870 tablet_bootstrap.cc:492] T 55b0ee0531e14410aa1fce3360b4e715 P 2f74a0910ea94f6aa814996cc6eff5be: Bootstrap starting.
I20260926 06:15:24.851199 9870 tablet_bootstrap.cc:654] T 55b0ee0531e14410aa1fce3360b4e715 P 2f74a0910ea94f6aa814996cc6eff5be: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:24.851400 9869 tablet_bootstrap.cc:654] T 55b0ee0531e14410aa1fce3360b4e715 P 8217ae7c8e30427aa59ee96d67f4bf91: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:24.851550 9870 log.cc:824] T 55b0ee0531e14410aa1fce3360b4e715 P 2f74a0910ea94f6aa814996cc6eff5be: Log is configured to *not* fsync() on all Append() calls
I20260926 06:15:24.851822 9869 log.cc:824] T 55b0ee0531e14410aa1fce3360b4e715 P 8217ae7c8e30427aa59ee96d67f4bf91: Log is configured to *not* fsync() on all Append() calls
I20260926 06:15:24.851917 9871 tablet_bootstrap.cc:492] T 55b0ee0531e14410aa1fce3360b4e715 P 96a7adaf68cd451ba1388e6550cf780c: Bootstrap starting.
I20260926 06:15:24.852527 9871 tablet_bootstrap.cc:654] T 55b0ee0531e14410aa1fce3360b4e715 P 96a7adaf68cd451ba1388e6550cf780c: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:24.852871 9871 log.cc:824] T 55b0ee0531e14410aa1fce3360b4e715 P 96a7adaf68cd451ba1388e6550cf780c: Log is configured to *not* fsync() on all Append() calls
I20260926 06:15:24.853170 9869 tablet_bootstrap.cc:492] T 55b0ee0531e14410aa1fce3360b4e715 P 8217ae7c8e30427aa59ee96d67f4bf91: No bootstrap required, opened a new log
I20260926 06:15:24.853266 9869 ts_tablet_manager.cc:1397] T 55b0ee0531e14410aa1fce3360b4e715 P 8217ae7c8e30427aa59ee96d67f4bf91: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 06:15:24.853550 9870 tablet_bootstrap.cc:492] T 55b0ee0531e14410aa1fce3360b4e715 P 2f74a0910ea94f6aa814996cc6eff5be: No bootstrap required, opened a new log
I20260926 06:15:24.853631 9870 ts_tablet_manager.cc:1397] T 55b0ee0531e14410aa1fce3360b4e715 P 2f74a0910ea94f6aa814996cc6eff5be: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 06:15:24.853941 9871 tablet_bootstrap.cc:492] T 55b0ee0531e14410aa1fce3360b4e715 P 96a7adaf68cd451ba1388e6550cf780c: No bootstrap required, opened a new log
I20260926 06:15:24.854048 9871 ts_tablet_manager.cc:1397] T 55b0ee0531e14410aa1fce3360b4e715 P 96a7adaf68cd451ba1388e6550cf780c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:15:24.855392 9869 raft_consensus.cc:348] T 55b0ee0531e14410aa1fce3360b4e715 P 8217ae7c8e30427aa59ee96d67f4bf91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f74a0910ea94f6aa814996cc6eff5be" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45563 } } peers { permanent_uuid: "96a7adaf68cd451ba1388e6550cf780c" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 38407 } } peers { permanent_uuid: "8217ae7c8e30427aa59ee96d67f4bf91" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 40801 } }
I20260926 06:15:24.855592 9869 raft_consensus.cc:374] T 55b0ee0531e14410aa1fce3360b4e715 P 8217ae7c8e30427aa59ee96d67f4bf91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:24.855641 9869 raft_consensus.cc:729] T 55b0ee0531e14410aa1fce3360b4e715 P 8217ae7c8e30427aa59ee96d67f4bf91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8217ae7c8e30427aa59ee96d67f4bf91, State: Initialized, Role: FOLLOWER
I20260926 06:15:24.855788 9869 consensus_queue.cc:260] T 55b0ee0531e14410aa1fce3360b4e715 P 8217ae7c8e30427aa59ee96d67f4bf91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f74a0910ea94f6aa814996cc6eff5be" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45563 } } peers { permanent_uuid: "96a7adaf68cd451ba1388e6550cf780c" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 38407 } } peers { permanent_uuid: "8217ae7c8e30427aa59ee96d67f4bf91" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 40801 } }
I20260926 06:15:24.856246 9568 heartbeater.cc:503] Master 127.2.110.254:46749 was elected leader, sending a full tablet report...
I20260926 06:15:24.856544 9869 ts_tablet_manager.cc:1428] T 55b0ee0531e14410aa1fce3360b4e715 P 8217ae7c8e30427aa59ee96d67f4bf91: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 06:15:24.856822 9871 raft_consensus.cc:348] T 55b0ee0531e14410aa1fce3360b4e715 P 96a7adaf68cd451ba1388e6550cf780c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f74a0910ea94f6aa814996cc6eff5be" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45563 } } peers { permanent_uuid: "96a7adaf68cd451ba1388e6550cf780c" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 38407 } } peers { permanent_uuid: "8217ae7c8e30427aa59ee96d67f4bf91" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 40801 } }
I20260926 06:15:24.856966 9871 raft_consensus.cc:374] T 55b0ee0531e14410aa1fce3360b4e715 P 96a7adaf68cd451ba1388e6550cf780c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:24.857003 9871 raft_consensus.cc:729] T 55b0ee0531e14410aa1fce3360b4e715 P 96a7adaf68cd451ba1388e6550cf780c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96a7adaf68cd451ba1388e6550cf780c, State: Initialized, Role: FOLLOWER
I20260926 06:15:24.857116 9871 consensus_queue.cc:260] T 55b0ee0531e14410aa1fce3360b4e715 P 96a7adaf68cd451ba1388e6550cf780c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f74a0910ea94f6aa814996cc6eff5be" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45563 } } peers { permanent_uuid: "96a7adaf68cd451ba1388e6550cf780c" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 38407 } } peers { permanent_uuid: "8217ae7c8e30427aa59ee96d67f4bf91" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 40801 } }
I20260926 06:15:24.857431 9871 ts_tablet_manager.cc:1428] T 55b0ee0531e14410aa1fce3360b4e715 P 96a7adaf68cd451ba1388e6550cf780c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 06:15:24.857543 9841 heartbeater.cc:503] Master 127.2.110.254:46749 was elected leader, sending a full tablet report...
I20260926 06:15:24.858700 9870 raft_consensus.cc:348] T 55b0ee0531e14410aa1fce3360b4e715 P 2f74a0910ea94f6aa814996cc6eff5be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f74a0910ea94f6aa814996cc6eff5be" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45563 } } peers { permanent_uuid: "96a7adaf68cd451ba1388e6550cf780c" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 38407 } } peers { permanent_uuid: "8217ae7c8e30427aa59ee96d67f4bf91" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 40801 } }
I20260926 06:15:24.858839 9870 raft_consensus.cc:374] T 55b0ee0531e14410aa1fce3360b4e715 P 2f74a0910ea94f6aa814996cc6eff5be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:24.858860 9870 raft_consensus.cc:729] T 55b0ee0531e14410aa1fce3360b4e715 P 2f74a0910ea94f6aa814996cc6eff5be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f74a0910ea94f6aa814996cc6eff5be, State: Initialized, Role: FOLLOWER
I20260926 06:15:24.858973 9870 consensus_queue.cc:260] T 55b0ee0531e14410aa1fce3360b4e715 P 2f74a0910ea94f6aa814996cc6eff5be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f74a0910ea94f6aa814996cc6eff5be" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45563 } } peers { permanent_uuid: "96a7adaf68cd451ba1388e6550cf780c" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 38407 } } peers { permanent_uuid: "8217ae7c8e30427aa59ee96d67f4bf91" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 40801 } }
I20260926 06:15:24.859202 9870 ts_tablet_manager.cc:1428] T 55b0ee0531e14410aa1fce3360b4e715 P 2f74a0910ea94f6aa814996cc6eff5be: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260926 06:15:24.859283 9705 heartbeater.cc:503] Master 127.2.110.254:46749 was elected leader, sending a full tablet report...
W20260926 06:15:24.918862 9707 tablet.cc:2369] T 55b0ee0531e14410aa1fce3360b4e715 P 2f74a0910ea94f6aa814996cc6eff5be: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:15:24.934367 9875 raft_consensus.cc:482] T 55b0ee0531e14410aa1fce3360b4e715 P 8217ae7c8e30427aa59ee96d67f4bf91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:15:24.934521 9875 raft_consensus.cc:504] T 55b0ee0531e14410aa1fce3360b4e715 P 8217ae7c8e30427aa59ee96d67f4bf91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f74a0910ea94f6aa814996cc6eff5be" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45563 } } peers { permanent_uuid: "96a7adaf68cd451ba1388e6550cf780c" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 38407 } } peers { permanent_uuid: "8217ae7c8e30427aa59ee96d67f4bf91" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 40801 } }
I20260926 06:15:24.935034 9875 leader_election.cc:288] T 55b0ee0531e14410aa1fce3360b4e715 P 8217ae7c8e30427aa59ee96d67f4bf91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f74a0910ea94f6aa814996cc6eff5be (127.2.110.194:45563), 96a7adaf68cd451ba1388e6550cf780c (127.2.110.195:38407)
I20260926 06:15:24.938275 9865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.935491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:35753 (local address 127.2.110.195:38407)
0926 06:15:24.935548 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:24.935550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:24.935559 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:15:24.935587 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:24.935588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:24.935597 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:24.935673 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:15:24.935863 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.936370 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.937428 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.937877 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.937985 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.937994 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:24.938161 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 06:15:24.938199 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 06:15:24.938282 9868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.935362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:46909 (local address 127.2.110.194:45563)
0926 06:15:24.935419 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:24.935422 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:24.935436 (+ 14us) server_negotiation.cc:407] Connection header received
0926 06:15:24.935503 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:24.935505 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:24.935517 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:24.935581 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:15:24.935763 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.936307 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.937461 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.937870 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.937977 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.937988 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:24.938153 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 06:15:24.938195 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 06:15:24.938315 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.935386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.195:38407 (local address 127.2.110.193:35753)
0926 06:15:24.935438 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:24.935449 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:24.935491 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:24.935684 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:24.935684 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:24.935686 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:15:24.935782 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:24.935784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.936439 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.936440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:24.937278 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:24.937281 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.937982 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.937984 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:24.938067 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.938071 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:15:24.938154 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 06:15:24.938198 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":4}
I20260926 06:15:24.938331 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.935264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.194:45563 (local address 127.2.110.193:46909)
0926 06:15:24.935331 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:24.935346 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:24.935432 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:24.935584 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:24.935585 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:24.935586 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:15:24.935681 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:24.935684 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.936409 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.936410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:24.937280 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:24.937282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.937977 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.937978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:24.938065 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.938070 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 06:15:24.938147 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 06:15:24.938195 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":20}
I20260926 06:15:24.938716 9658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55b0ee0531e14410aa1fce3360b4e715" candidate_uuid: "8217ae7c8e30427aa59ee96d67f4bf91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f74a0910ea94f6aa814996cc6eff5be" is_pre_election: true
I20260926 06:15:24.938916 9658 raft_consensus.cc:2454] T 55b0ee0531e14410aa1fce3360b4e715 P 2f74a0910ea94f6aa814996cc6eff5be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8217ae7c8e30427aa59ee96d67f4bf91 in term 0.
I20260926 06:15:24.938956 9794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55b0ee0531e14410aa1fce3360b4e715" candidate_uuid: "8217ae7c8e30427aa59ee96d67f4bf91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96a7adaf68cd451ba1388e6550cf780c" is_pre_election: true
I20260926 06:15:24.939080 9794 raft_consensus.cc:2454] T 55b0ee0531e14410aa1fce3360b4e715 P 96a7adaf68cd451ba1388e6550cf780c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8217ae7c8e30427aa59ee96d67f4bf91 in term 0.
I20260926 06:15:24.939213 9455 leader_election.cc:302] T 55b0ee0531e14410aa1fce3360b4e715 P 8217ae7c8e30427aa59ee96d67f4bf91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f74a0910ea94f6aa814996cc6eff5be, 8217ae7c8e30427aa59ee96d67f4bf91; no voters:
I20260926 06:15:24.939424 9875 raft_consensus.cc:2781] T 55b0ee0531e14410aa1fce3360b4e715 P 8217ae7c8e30427aa59ee96d67f4bf91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:15:24.939510 9875 raft_consensus.cc:482] T 55b0ee0531e14410aa1fce3360b4e715 P 8217ae7c8e30427aa59ee96d67f4bf91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:15:24.939553 9875 raft_consensus.cc:3037] T 55b0ee0531e14410aa1fce3360b4e715 P 8217ae7c8e30427aa59ee96d67f4bf91 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:24.940331 9875 raft_consensus.cc:504] T 55b0ee0531e14410aa1fce3360b4e715 P 8217ae7c8e30427aa59ee96d67f4bf91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f74a0910ea94f6aa814996cc6eff5be" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45563 } } peers { permanent_uuid: "96a7adaf68cd451ba1388e6550cf780c" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 38407 } } peers { permanent_uuid: "8217ae7c8e30427aa59ee96d67f4bf91" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 40801 } }
I20260926 06:15:24.940495 9875 leader_election.cc:288] T 55b0ee0531e14410aa1fce3360b4e715 P 8217ae7c8e30427aa59ee96d67f4bf91 [CANDIDATE]: Term 1 election: Requested vote from peers 2f74a0910ea94f6aa814996cc6eff5be (127.2.110.194:45563), 96a7adaf68cd451ba1388e6550cf780c (127.2.110.195:38407)
I20260926 06:15:24.940783 9658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55b0ee0531e14410aa1fce3360b4e715" candidate_uuid: "8217ae7c8e30427aa59ee96d67f4bf91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f74a0910ea94f6aa814996cc6eff5be"
I20260926 06:15:24.940843 9794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55b0ee0531e14410aa1fce3360b4e715" candidate_uuid: "8217ae7c8e30427aa59ee96d67f4bf91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96a7adaf68cd451ba1388e6550cf780c"
I20260926 06:15:24.940924 9794 raft_consensus.cc:3037] T 55b0ee0531e14410aa1fce3360b4e715 P 96a7adaf68cd451ba1388e6550cf780c [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:24.940924 9658 raft_consensus.cc:3037] T 55b0ee0531e14410aa1fce3360b4e715 P 2f74a0910ea94f6aa814996cc6eff5be [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:24.941874 9658 raft_consensus.cc:2454] T 55b0ee0531e14410aa1fce3360b4e715 P 2f74a0910ea94f6aa814996cc6eff5be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8217ae7c8e30427aa59ee96d67f4bf91 in term 1.
I20260926 06:15:24.941874 9794 raft_consensus.cc:2454] T 55b0ee0531e14410aa1fce3360b4e715 P 96a7adaf68cd451ba1388e6550cf780c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8217ae7c8e30427aa59ee96d67f4bf91 in term 1.
I20260926 06:15:24.942104 9455 leader_election.cc:302] T 55b0ee0531e14410aa1fce3360b4e715 P 8217ae7c8e30427aa59ee96d67f4bf91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f74a0910ea94f6aa814996cc6eff5be, 8217ae7c8e30427aa59ee96d67f4bf91; no voters:
I20260926 06:15:24.942222 9875 raft_consensus.cc:2781] T 55b0ee0531e14410aa1fce3360b4e715 P 8217ae7c8e30427aa59ee96d67f4bf91 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:15:24.942391 9875 raft_consensus.cc:686] T 55b0ee0531e14410aa1fce3360b4e715 P 8217ae7c8e30427aa59ee96d67f4bf91 [term 1 LEADER]: Becoming Leader. State: Replica: 8217ae7c8e30427aa59ee96d67f4bf91, State: Running, Role: LEADER
I20260926 06:15:24.942785 9875 consensus_queue.cc:237] T 55b0ee0531e14410aa1fce3360b4e715 P 8217ae7c8e30427aa59ee96d67f4bf91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f74a0910ea94f6aa814996cc6eff5be" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45563 } } peers { permanent_uuid: "96a7adaf68cd451ba1388e6550cf780c" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 38407 } } peers { permanent_uuid: "8217ae7c8e30427aa59ee96d67f4bf91" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 40801 } }
I20260926 06:15:24.943847 8597 catalog_manager.cc:5495] T 55b0ee0531e14410aa1fce3360b4e715 P 8217ae7c8e30427aa59ee96d67f4bf91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8217ae7c8e30427aa59ee96d67f4bf91 (127.2.110.193). New cstate: current_term: 1 leader_uuid: "8217ae7c8e30427aa59ee96d67f4bf91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f74a0910ea94f6aa814996cc6eff5be" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96a7adaf68cd451ba1388e6550cf780c" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 38407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8217ae7c8e30427aa59ee96d67f4bf91" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 40801 } health_report { overall_health: HEALTHY } } }
I20260926 06:15:24.967465 9867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.965030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41524 (local address 127.2.110.193:40801)
0926 06:15:24.965112 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:24.965114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:24.965124 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:15:24.965167 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:24.965168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:24.965204 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:24.965300 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:15:24.965485 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.965954 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.966709 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.966904 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.966969 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.967101 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:15:24.967208 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:15:24.967288 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:24.967319 (+ 31us) server_negotiation.cc:299] Negotiation successful
0926 06:15:24.967340 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 06:15:24.969504 9658 raft_consensus.cc:1261] T 55b0ee0531e14410aa1fce3360b4e715 P 2f74a0910ea94f6aa814996cc6eff5be [term 1 FOLLOWER]: Refusing update from remote peer 8217ae7c8e30427aa59ee96d67f4bf91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:15:24.969504 9794 raft_consensus.cc:1261] T 55b0ee0531e14410aa1fce3360b4e715 P 96a7adaf68cd451ba1388e6550cf780c [term 1 FOLLOWER]: Refusing update from remote peer 8217ae7c8e30427aa59ee96d67f4bf91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:15:24.969879 9878 consensus_queue.cc:1035] T 55b0ee0531e14410aa1fce3360b4e715 P 8217ae7c8e30427aa59ee96d67f4bf91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f74a0910ea94f6aa814996cc6eff5be" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:15:24.970031 9875 consensus_queue.cc:1035] T 55b0ee0531e14410aa1fce3360b4e715 P 8217ae7c8e30427aa59ee96d67f4bf91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96a7adaf68cd451ba1388e6550cf780c" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 38407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:15:24.973194 9882 mvcc.cc:205] Tried to move back new op lower bound from 7333492019071873024 to 7333492018966470656. Current Snapshot: MvccSnapshot[applied={T|T < 7333492019071873024}]
I20260926 06:15:24.973279 9883 mvcc.cc:205] Tried to move back new op lower bound from 7333492019071873024 to 7333492018966470656. Current Snapshot: MvccSnapshot[applied={T|T < 7333492019071873024}]
I20260926 06:15:24.974031 9881 mvcc.cc:205] Tried to move back new op lower bound from 7333492019071873024 to 7333492018966470656. Current Snapshot: MvccSnapshot[applied={T|T < 7333492019071873024 or (T in {7333492019071873024})}]
I20260926 06:15:24.975602 8598 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:15:24.975701 8598 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:15:24.977290 8598 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eed8ee75c5394469994882db1a963644: Sending DeleteTablet for 3 replicas of tablet 55b0ee0531e14410aa1fce3360b4e715
I20260926 06:15:24.977788 9774 tablet_service.cc:1514] Processing DeleteTablet for tablet 55b0ee0531e14410aa1fce3360b4e715 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:15:24 UTC) from {username='oryx', principal='oryx/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:38414
I20260926 06:15:24.977880 9638 tablet_service.cc:1514] Processing DeleteTablet for tablet 55b0ee0531e14410aa1fce3360b4e715 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:15:24 UTC) from {username='oryx', principal='oryx/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:37924
I20260926 06:15:24.978051 9773 tablet_service.cc:1514] Processing DeleteTablet for tablet 55b0ee0531e14410aa1fce3360b4e715 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:15:24 UTC) from {username='oryx', principal='oryx/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:38414
I20260926 06:15:24.978152 9501 tablet_service.cc:1514] Processing DeleteTablet for tablet 55b0ee0531e14410aa1fce3360b4e715 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:15:24 UTC) from {username='oryx', principal='oryx/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:41510
I20260926 06:15:24.978356 9892 tablet_replica.cc:330] T 55b0ee0531e14410aa1fce3360b4e715 P 2f74a0910ea94f6aa814996cc6eff5be: stopping tablet replica
I20260926 06:15:24.978457 9892 raft_consensus.cc:2229] T 55b0ee0531e14410aa1fce3360b4e715 P 2f74a0910ea94f6aa814996cc6eff5be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:15:24.978550 9892 raft_consensus.cc:2258] T 55b0ee0531e14410aa1fce3360b4e715 P 2f74a0910ea94f6aa814996cc6eff5be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:15:24.978693 9894 tablet_replica.cc:330] T 55b0ee0531e14410aa1fce3360b4e715 P 8217ae7c8e30427aa59ee96d67f4bf91: stopping tablet replica
I20260926 06:15:24.978785 9894 raft_consensus.cc:2229] T 55b0ee0531e14410aa1fce3360b4e715 P 8217ae7c8e30427aa59ee96d67f4bf91 [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:15:24.979247 9892 ts_tablet_manager.cc:1905] T 55b0ee0531e14410aa1fce3360b4e715 P 2f74a0910ea94f6aa814996cc6eff5be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:15:24.979261 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 8566
I20260926 06:15:24.980927 9894 raft_consensus.cc:2258] T 55b0ee0531e14410aa1fce3360b4e715 P 8217ae7c8e30427aa59ee96d67f4bf91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:15:24.981626 9892 ts_tablet_manager.cc:1918] T 55b0ee0531e14410aa1fce3360b4e715 P 2f74a0910ea94f6aa814996cc6eff5be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:15:24.981709 9892 log.cc:1196] T 55b0ee0531e14410aa1fce3360b4e715 P 2f74a0910ea94f6aa814996cc6eff5be: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-1/wal/wals/55b0ee0531e14410aa1fce3360b4e715
I20260926 06:15:24.981988 9892 ts_tablet_manager.cc:1939] T 55b0ee0531e14410aa1fce3360b4e715 P 2f74a0910ea94f6aa814996cc6eff5be: Deleting consensus metadata
I20260926 06:15:24.986232 9853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.978534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35306 (local address 127.2.110.252:41235)
0926 06:15:24.978573 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:24.978575 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:24.978584 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:15:24.978605 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:24.978606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:24.978617 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:24.978677 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:24.979854 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.980641 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.981720 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.982045 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.982094 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.983513 (+ 1419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:24.983514 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:24.983514 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:24.983521 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:24.984972 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:24.985376 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:24.985378 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:24.985378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:24.985441 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:24.985645 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:24.985645 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:24.985646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:24.985784 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:24.985844 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:24.986026 (+ 182us) server_negotiation.cc:299] Negotiation successful
0926 06:15:24.986118 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260926 06:15:24.986970 9593 connection.cc:535] server connection from 127.0.0.1:37924 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 06:15:24.987924 9894 ts_tablet_manager.cc:1905] T 55b0ee0531e14410aa1fce3360b4e715 P 8217ae7c8e30427aa59ee96d67f4bf91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:15:24.989701 9894 ts_tablet_manager.cc:1918] T 55b0ee0531e14410aa1fce3360b4e715 P 8217ae7c8e30427aa59ee96d67f4bf91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:15:24.989791 9894 log.cc:1196] T 55b0ee0531e14410aa1fce3360b4e715 P 8217ae7c8e30427aa59ee96d67f4bf91: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-0/wal/wals/55b0ee0531e14410aa1fce3360b4e715
I20260926 06:15:24.990114 9894 ts_tablet_manager.cc:1939] T 55b0ee0531e14410aa1fce3360b4e715 P 8217ae7c8e30427aa59ee96d67f4bf91: Deleting consensus metadata
W20260926 06:15:24.990497 9454 connection.cc:441] server connection from 127.0.0.1:41510 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41510 (request call id 1) could send its response
I20260926 06:15:24.990691 9891 tablet_replica.cc:330] T 55b0ee0531e14410aa1fce3360b4e715 P 96a7adaf68cd451ba1388e6550cf780c: stopping tablet replica
I20260926 06:15:24.990824 9891 raft_consensus.cc:2229] T 55b0ee0531e14410aa1fce3360b4e715 P 96a7adaf68cd451ba1388e6550cf780c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:15:24.990984 9891 raft_consensus.cc:2258] T 55b0ee0531e14410aa1fce3360b4e715 P 96a7adaf68cd451ba1388e6550cf780c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:15:24.992146 9891 ts_tablet_manager.cc:1905] T 55b0ee0531e14410aa1fce3360b4e715 P 96a7adaf68cd451ba1388e6550cf780c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:15:24.993965 9891 ts_tablet_manager.cc:1918] T 55b0ee0531e14410aa1fce3360b4e715 P 96a7adaf68cd451ba1388e6550cf780c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:15:24.994097 9891 log.cc:1196] T 55b0ee0531e14410aa1fce3360b4e715 P 96a7adaf68cd451ba1388e6550cf780c: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-2/wal/wals/55b0ee0531e14410aa1fce3360b4e715
I20260926 06:15:24.994479 9891 ts_tablet_manager.cc:1939] T 55b0ee0531e14410aa1fce3360b4e715 P 96a7adaf68cd451ba1388e6550cf780c: Deleting consensus metadata
W20260926 06:15:24.994899 9730 connection.cc:441] server connection from 127.0.0.1:38414 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38414 (request call id 1) could send its response
W20260926 06:15:24.996376 9730 connection.cc:441] server connection from 127.0.0.1:38414 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38414 (request call id 2) could send its response
I20260926 06:15:24.998994 9853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.989325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35320 (local address 127.2.110.252:41235)
0926 06:15:24.989398 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:24.989400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:24.989540 (+ 140us) server_negotiation.cc:407] Connection header received
0926 06:15:24.989640 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:24.989642 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:24.989653 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:24.989685 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:24.990850 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.991726 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.993126 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.993483 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.993526 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.995448 (+ 1922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:24.995449 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:24.995449 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:24.995457 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:24.997643 (+ 2186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:24.998082 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:24.998083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:24.998084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:24.998144 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:24.998390 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:24.998391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:24.998392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:24.998536 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:24.998608 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:24.998811 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 06:15:24.998895 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 06:15:24.999169 9247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:24.989053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54278 (local address 127.2.110.253:35513)
0926 06:15:24.989106 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:24.989109 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:24.989438 (+ 329us) server_negotiation.cc:407] Connection header received
0926 06:15:24.989569 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:24.989571 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:24.989584 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:24.989622 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:24.990995 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.991787 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.993036 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:24.993384 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:24.993439 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:24.995953 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:24.995955 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:24.995955 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:24.995964 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:24.997856 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:24.998299 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:24.998300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:24.998301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:24.998355 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:24.998603 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:24.998604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:24.998604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:24.998731 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:24.998844 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:24.998970 (+ 126us) server_negotiation.cc:299] Negotiation successful
0926 06:15:24.999055 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 06:15:25.974949 9902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:25.974683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:46749 (local address 127.2.110.193:45151)
0926 06:15:25.974819 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:25.974897 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:46749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":65,"threads_started":1}
W20260926 06:15:25.975184 9568 heartbeater.cc:644] Failed to heartbeat to 127.2.110.254:46749 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.110.254:46749: connect: Connection refused (error 111)
I20260926 06:15:25.976181 9903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:25.975900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:46749 (local address 127.2.110.194:41511)
0926 06:15:25.976027 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:25.976128 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:46749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":73,"threads_started":1}
W20260926 06:15:25.976523 9705 heartbeater.cc:644] Failed to heartbeat to 127.2.110.254:46749 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.110.254:46749: connect: Connection refused (error 111)
I20260926 06:15:25.977994 9904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:25.977785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:46749 (local address 127.2.110.195:33817)
0926 06:15:25.977896 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:25.977957 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:46749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":50,"threads_started":1}
W20260926 06:15:25.978348 9841 heartbeater.cc:644] Failed to heartbeat to 127.2.110.254:46749 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.110.254:46749: connect: Connection refused (error 111)
I20260926 06:15:26.477725 9905 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eed8ee75c5394469994882db1a963644)
I20260926 06:15:26.477864 9905 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } peers { permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35513 } attrs { promote: false } } peers { permanent_uuid: "f5662db075fb4054a7d7380e5834eb1f" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 41235 } attrs { promote: false } }
I20260926 06:15:26.477972 9906 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eed8ee75c5394469994882db1a963644)
I20260926 06:15:26.478096 9906 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } peers { permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35513 } attrs { promote: false } } peers { permanent_uuid: "f5662db075fb4054a7d7380e5834eb1f" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 41235 } attrs { promote: false } }
I20260926 06:15:26.478411 9906 leader_election.cc:288] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eed8ee75c5394469994882db1a963644 (127.2.110.254:46749), f5662db075fb4054a7d7380e5834eb1f (127.2.110.252:41235)
I20260926 06:15:26.478641 9907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:26.478375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:46749 (local address 127.0.0.1:44908)
0926 06:15:26.478535 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:26.478589 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:46749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":39,"threads_started":1}
I20260926 06:15:26.478838 9908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:26.478516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:46749 (local address 127.0.0.1:44924)
0926 06:15:26.478613 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:26.478663 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:46749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77,"thread_start_us":50,"threads_started":1}
W20260926 06:15:26.478894 9373 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.110.254:46749: connect: Connection refused (error 111)
I20260926 06:15:26.479028 9905 leader_election.cc:288] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eed8ee75c5394469994882db1a963644 (127.2.110.254:46749), 4a5a9e72e816463b9cfcb45b2f858088 (127.2.110.253:35513)
W20260926 06:15:26.479033 9181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.110.254:46749: connect: Connection refused (error 111)
I20260926 06:15:26.479456 9908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:26.479332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:46749 (local address 127.0.0.1:44944)
0926 06:15:26.479394 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:26.479425 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:46749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 06:15:26.479460 9909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:26.479224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:46749 (local address 127.0.0.1:44936)
0926 06:15:26.479395 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:26.479432 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:46749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
W20260926 06:15:26.479526 9373 leader_election.cc:334] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer eed8ee75c5394469994882db1a963644 (127.2.110.254:46749): Network error: Client connection negotiation failed: client connection to 127.2.110.254:46749: connect: Connection refused (error 111)
W20260926 06:15:26.479606 9181 leader_election.cc:334] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer eed8ee75c5394469994882db1a963644 (127.2.110.254:46749): Network error: Client connection negotiation failed: client connection to 127.2.110.254:46749: connect: Connection refused (error 111)
I20260926 06:15:26.481954 9910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:26.478673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.252:41235 (local address 127.0.0.1:53854)
0926 06:15:26.478825 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:26.478834 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:26.478889 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:26.479066 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:26.479067 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:26.479069 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:15:26.479126 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:26.479128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:26.480078 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:26.480079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:26.481012 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:26.481015 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:26.481689 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:26.481691 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:26.481783 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:26.481787 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:15:26.481851 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 06:15:26.481904 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":51,"threads_started":1}
I20260926 06:15:26.482086 9912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:26.478776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53854 (local address 127.2.110.252:41235)
0926 06:15:26.478898 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:26.478900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:26.478909 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:15:26.478942 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:26.478943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:26.478953 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:26.479014 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:15:26.479213 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:26.479983 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:26.481099 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:26.481565 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:26.481674 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:26.481688 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:26.481867 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 06:15:26.481941 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":46,"threads_started":1}
I20260926 06:15:26.482335 9398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4a5a9e72e816463b9cfcb45b2f858088" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f5662db075fb4054a7d7380e5834eb1f" is_pre_election: true
I20260926 06:15:26.482463 9398 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a5a9e72e816463b9cfcb45b2f858088 in term 1.
I20260926 06:15:26.482625 9181 leader_election.cc:302] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a5a9e72e816463b9cfcb45b2f858088, f5662db075fb4054a7d7380e5834eb1f; no voters: eed8ee75c5394469994882db1a963644
I20260926 06:15:26.482757 9906 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 06:15:26.482801 9906 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088 [term 1 FOLLOWER]: Starting leader election (detected failure of leader eed8ee75c5394469994882db1a963644)
I20260926 06:15:26.482831 9906 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088 [term 1 FOLLOWER]: Advancing to term 2
I20260926 06:15:26.482915 9907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:26.478591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.253:35513 (local address 127.0.0.1:35906)
0926 06:15:26.478716 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:26.478727 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:26.478789 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:26.479461 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:26.479462 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:26.479465 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:15:26.479645 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:26.479647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:26.480368 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:26.480369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:26.481672 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:26.481680 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:26.482612 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:26.482612 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:26.482711 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:26.482718 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 06:15:26.482771 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 06:15:26.482830 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":37,"threads_started":1}
I20260926 06:15:26.483140 9911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:26.478751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35906 (local address 127.2.110.253:35513)
0926 06:15:26.479306 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:26.479308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:26.479318 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:15:26.479341 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:26.479342 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:26.479357 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:26.479766 (+ 409us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:15:26.479800 (+ 34us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:26.480261 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:26.481828 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:26.482517 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:26.482610 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:26.482622 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:26.483036 (+ 414us) server_negotiation.cc:299] Negotiation successful
0926 06:15:26.483082 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":36,"threads_started":1}
I20260926 06:15:26.483350 9206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f5662db075fb4054a7d7380e5834eb1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4a5a9e72e816463b9cfcb45b2f858088" is_pre_election: true
I20260926 06:15:26.483857 9906 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } peers { permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35513 } attrs { promote: false } } peers { permanent_uuid: "f5662db075fb4054a7d7380e5834eb1f" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 41235 } attrs { promote: false } }
I20260926 06:15:26.484066 9206 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f5662db075fb4054a7d7380e5834eb1f in current term 2: Already voted for candidate 4a5a9e72e816463b9cfcb45b2f858088 in this term.
I20260926 06:15:26.484108 9906 leader_election.cc:288] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088 [CANDIDATE]: Term 2 election: Requested vote from peers eed8ee75c5394469994882db1a963644 (127.2.110.254:46749), f5662db075fb4054a7d7380e5834eb1f (127.2.110.252:41235)
I20260926 06:15:26.484436 9910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:26.484262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:46749 (local address 127.0.0.1:44952)
0926 06:15:26.484370 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:26.484411 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:46749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 06:15:26.484436 9374 leader_election.cc:302] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f5662db075fb4054a7d7380e5834eb1f; no voters: 4a5a9e72e816463b9cfcb45b2f858088, eed8ee75c5394469994882db1a963644
I20260926 06:15:26.484619 9398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4a5a9e72e816463b9cfcb45b2f858088" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f5662db075fb4054a7d7380e5834eb1f"
I20260926 06:15:26.484692 9398 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f [term 1 FOLLOWER]: Advancing to term 2
I20260926 06:15:26.484907 9910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:26.484789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:46749 (local address 127.0.0.1:44964)
0926 06:15:26.484843 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:26.484879 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:46749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 06:15:26.484993 9181 leader_election.cc:334] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer eed8ee75c5394469994882db1a963644 (127.2.110.254:46749): Network error: Client connection negotiation failed: client connection to 127.2.110.254:46749: connect: Connection refused (error 111)
I20260926 06:15:26.485510 9398 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a5a9e72e816463b9cfcb45b2f858088 in term 2.
I20260926 06:15:26.485596 9905 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 06:15:26.485710 9181 leader_election.cc:302] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a5a9e72e816463b9cfcb45b2f858088, f5662db075fb4054a7d7380e5834eb1f; no voters: eed8ee75c5394469994882db1a963644
I20260926 06:15:26.485818 9906 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 06:15:26.485986 9906 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088 [term 2 LEADER]: Becoming Leader. State: Replica: 4a5a9e72e816463b9cfcb45b2f858088, State: Running, Role: LEADER
I20260926 06:15:26.486078 9906 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } peers { permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35513 } attrs { promote: false } } peers { permanent_uuid: "f5662db075fb4054a7d7380e5834eb1f" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 41235 } attrs { promote: false } }
I20260926 06:15:26.486495 9915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a5a9e72e816463b9cfcb45b2f858088. Latest consensus state: current_term: 2 leader_uuid: "4a5a9e72e816463b9cfcb45b2f858088" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } peers { permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35513 } attrs { promote: false } } peers { permanent_uuid: "f5662db075fb4054a7d7380e5834eb1f" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 41235 } attrs { promote: false } } }
I20260926 06:15:26.486593 9915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088 [sys.catalog]: This master's current role is: LEADER
I20260926 06:15:26.486801 9917 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:15:26.487257 9917 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:15:26.487398 9917 catalog_manager.cc:1259] Loaded cluster ID: b50e97be6eac4bccbae81dfbad9212a1
I20260926 06:15:26.487437 9917 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:15:26.487588 9917 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:15:26.487862 9917 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088: Loaded TSK: 0
I20260926 06:15:26.488044 9917 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:15:26.496501 9195 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54278:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:15:26.498450 9195 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:15:26.499917 9910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:26.499789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:46749 (local address 127.0.0.1:44982)
0926 06:15:26.499848 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:26.499886 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:46749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 06:15:26.499984 9398 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f [term 2 FOLLOWER]: Refusing update from remote peer 4a5a9e72e816463b9cfcb45b2f858088: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260926 06:15:26.500479 9910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:26.500369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:46749 (local address 127.0.0.1:44998)
0926 06:15:26.500432 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:26.500461 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:46749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 06:15:26.500535 9906 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5662db075fb4054a7d7380e5834eb1f" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 41235 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260926 06:15:26.500775 9181 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088 -> Peer eed8ee75c5394469994882db1a963644 (127.2.110.254:46749): Couldn't send request to peer eed8ee75c5394469994882db1a963644. Status: Network error: Client connection negotiation failed: client connection to 127.2.110.254:46749: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 06:15:26.501588 9905 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a5a9e72e816463b9cfcb45b2f858088. Latest consensus state: current_term: 2 leader_uuid: "4a5a9e72e816463b9cfcb45b2f858088" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } peers { permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35513 } attrs { promote: false } } peers { permanent_uuid: "f5662db075fb4054a7d7380e5834eb1f" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 41235 } attrs { promote: false } } }
I20260926 06:15:26.501816 9905 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 06:15:26.502436 9906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "4a5a9e72e816463b9cfcb45b2f858088" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } peers { permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35513 } attrs { promote: false } } peers { permanent_uuid: "f5662db075fb4054a7d7380e5834eb1f" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 41235 } attrs { promote: false } } }
I20260926 06:15:26.502682 9906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088 [sys.catalog]: This master's current role is: LEADER
I20260926 06:15:26.502925 9905 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "4a5a9e72e816463b9cfcb45b2f858088" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eed8ee75c5394469994882db1a963644" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46749 } } peers { permanent_uuid: "4a5a9e72e816463b9cfcb45b2f858088" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35513 } attrs { promote: false } } peers { permanent_uuid: "f5662db075fb4054a7d7380e5834eb1f" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 41235 } attrs { promote: false } } }
I20260926 06:15:26.503060 9905 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5662db075fb4054a7d7380e5834eb1f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 06:15:26.508919 9924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:26.505474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50484 (local address 127.2.110.193:40801)
0926 06:15:26.505598 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:26.505600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:26.505735 (+ 135us) server_negotiation.cc:407] Connection header received
0926 06:15:26.505838 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:26.505840 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:26.505857 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:26.505947 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:15:26.506141 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:26.506858 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:26.507959 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:26.508540 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:26.508623 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:26.508638 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:26.508796 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 06:15:26.508845 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":68,"threads_started":1}
I20260926 06:15:26.508917 9908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:26.505437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.193:40801 (local address 127.0.0.1:50484)
0926 06:15:26.505578 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:26.505590 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:26.505759 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:26.505969 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:26.505971 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:26.505974 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:15:26.506035 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:26.506036 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:26.506939 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:26.506940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:26.507861 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:26.507863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:26.508662 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:26.508663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:26.508718 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:26.508721 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:15:26.508791 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 06:15:26.508838 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123}
I20260926 06:15:26.509367 9501 tablet_service.cc:1467] Processing CreateTablet for tablet 092fd71863fb405ca247459e6c6c3c2b (DEFAULT_TABLE table=test-table [id=0d1cc70a5a854a4b9e20f9a92624c7d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:15:26.509510 9501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 092fd71863fb405ca247459e6c6c3c2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:26.509521 9910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:26.505611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.194:45563 (local address 127.0.0.1:45958)
0926 06:15:26.505742 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:26.505754 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:26.505798 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:26.506500 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:26.506501 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:26.506502 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:15:26.506540 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:26.506541 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:26.507510 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:26.507511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:26.508432 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:26.508434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:26.509251 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:26.509252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:26.509321 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:26.509325 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:15:26.509394 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 06:15:26.509458 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":91,"threads_started":1}
I20260926 06:15:26.509577 9926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:26.505897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60656 (local address 127.2.110.195:38407)
0926 06:15:26.506072 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:26.506074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:26.506084 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:15:26.506115 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:26.506116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:26.506129 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:26.506204 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:15:26.506409 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:26.507360 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:26.508625 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:26.509187 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:26.509250 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:26.509261 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:26.509450 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 06:15:26.509498 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":81,"threads_started":1}
I20260926 06:15:26.509788 9925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:26.505532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.195:38407 (local address 127.0.0.1:60656)
0926 06:15:26.505742 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:26.505754 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:26.505798 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:26.506277 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:26.506278 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:26.506280 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:15:26.506311 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:26.506312 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:26.507459 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:26.507460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:26.508545 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:26.508547 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:26.509288 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:26.509289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:26.509370 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:26.509373 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:15:26.509645 (+ 272us) client_negotiation.cc:240] Negotiation successful
0926 06:15:26.509732 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":32}
I20260926 06:15:26.509935 9927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:26.505891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45958 (local address 127.2.110.194:45563)
0926 06:15:26.506297 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:26.506299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:26.506307 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:15:26.506334 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:26.506335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:26.506346 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:26.506475 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:15:26.506688 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:26.507438 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:26.508519 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:26.509150 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:26.509704 (+ 554us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:26.509719 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:26.509738 (+ 19us) server_negotiation.cc:299] Negotiation successful
0926 06:15:26.509785 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":114,"threads_started":1}
I20260926 06:15:26.510144 9638 tablet_service.cc:1467] Processing CreateTablet for tablet 092fd71863fb405ca247459e6c6c3c2b (DEFAULT_TABLE table=test-table [id=0d1cc70a5a854a4b9e20f9a92624c7d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:15:26.510303 9638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 092fd71863fb405ca247459e6c6c3c2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:26.511061 9928 tablet_bootstrap.cc:492] T 092fd71863fb405ca247459e6c6c3c2b P 8217ae7c8e30427aa59ee96d67f4bf91: Bootstrap starting.
I20260926 06:15:26.511193 9773 tablet_service.cc:1467] Processing CreateTablet for tablet 092fd71863fb405ca247459e6c6c3c2b (DEFAULT_TABLE table=test-table [id=0d1cc70a5a854a4b9e20f9a92624c7d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:15:26.511310 9773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 092fd71863fb405ca247459e6c6c3c2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:26.511478 9928 tablet_bootstrap.cc:654] T 092fd71863fb405ca247459e6c6c3c2b P 8217ae7c8e30427aa59ee96d67f4bf91: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:26.511825 9929 tablet_bootstrap.cc:492] T 092fd71863fb405ca247459e6c6c3c2b P 2f74a0910ea94f6aa814996cc6eff5be: Bootstrap starting.
I20260926 06:15:26.512253 9928 tablet_bootstrap.cc:492] T 092fd71863fb405ca247459e6c6c3c2b P 8217ae7c8e30427aa59ee96d67f4bf91: No bootstrap required, opened a new log
I20260926 06:15:26.512329 9928 ts_tablet_manager.cc:1397] T 092fd71863fb405ca247459e6c6c3c2b P 8217ae7c8e30427aa59ee96d67f4bf91: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:26.512493 9929 tablet_bootstrap.cc:654] T 092fd71863fb405ca247459e6c6c3c2b P 2f74a0910ea94f6aa814996cc6eff5be: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:26.512471 9928 raft_consensus.cc:348] T 092fd71863fb405ca247459e6c6c3c2b P 8217ae7c8e30427aa59ee96d67f4bf91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8217ae7c8e30427aa59ee96d67f4bf91" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 40801 } } peers { permanent_uuid: "96a7adaf68cd451ba1388e6550cf780c" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 38407 } } peers { permanent_uuid: "2f74a0910ea94f6aa814996cc6eff5be" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45563 } }
I20260926 06:15:26.512569 9928 raft_consensus.cc:374] T 092fd71863fb405ca247459e6c6c3c2b P 8217ae7c8e30427aa59ee96d67f4bf91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:26.512598 9928 raft_consensus.cc:729] T 092fd71863fb405ca247459e6c6c3c2b P 8217ae7c8e30427aa59ee96d67f4bf91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8217ae7c8e30427aa59ee96d67f4bf91, State: Initialized, Role: FOLLOWER
I20260926 06:15:26.512662 9928 consensus_queue.cc:260] T 092fd71863fb405ca247459e6c6c3c2b P 8217ae7c8e30427aa59ee96d67f4bf91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8217ae7c8e30427aa59ee96d67f4bf91" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 40801 } } peers { permanent_uuid: "96a7adaf68cd451ba1388e6550cf780c" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 38407 } } peers { permanent_uuid: "2f74a0910ea94f6aa814996cc6eff5be" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45563 } }
I20260926 06:15:26.512813 9928 ts_tablet_manager.cc:1428] T 092fd71863fb405ca247459e6c6c3c2b P 8217ae7c8e30427aa59ee96d67f4bf91: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 06:15:26.513128 9929 tablet_bootstrap.cc:492] T 092fd71863fb405ca247459e6c6c3c2b P 2f74a0910ea94f6aa814996cc6eff5be: No bootstrap required, opened a new log
I20260926 06:15:26.513178 9929 ts_tablet_manager.cc:1397] T 092fd71863fb405ca247459e6c6c3c2b P 2f74a0910ea94f6aa814996cc6eff5be: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:26.513294 9929 raft_consensus.cc:348] T 092fd71863fb405ca247459e6c6c3c2b P 2f74a0910ea94f6aa814996cc6eff5be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8217ae7c8e30427aa59ee96d67f4bf91" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 40801 } } peers { permanent_uuid: "96a7adaf68cd451ba1388e6550cf780c" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 38407 } } peers { permanent_uuid: "2f74a0910ea94f6aa814996cc6eff5be" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45563 } }
I20260926 06:15:26.513392 9929 raft_consensus.cc:374] T 092fd71863fb405ca247459e6c6c3c2b P 2f74a0910ea94f6aa814996cc6eff5be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:26.513432 9929 raft_consensus.cc:729] T 092fd71863fb405ca247459e6c6c3c2b P 2f74a0910ea94f6aa814996cc6eff5be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f74a0910ea94f6aa814996cc6eff5be, State: Initialized, Role: FOLLOWER
I20260926 06:15:26.513477 9929 consensus_queue.cc:260] T 092fd71863fb405ca247459e6c6c3c2b P 2f74a0910ea94f6aa814996cc6eff5be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8217ae7c8e30427aa59ee96d67f4bf91" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 40801 } } peers { permanent_uuid: "96a7adaf68cd451ba1388e6550cf780c" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 38407 } } peers { permanent_uuid: "2f74a0910ea94f6aa814996cc6eff5be" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45563 } }
I20260926 06:15:26.513625 9929 ts_tablet_manager.cc:1428] T 092fd71863fb405ca247459e6c6c3c2b P 2f74a0910ea94f6aa814996cc6eff5be: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:26.513813 9569 heartbeater.cc:503] Master 127.2.110.253:35513 was elected leader, sending a full tablet report...
I20260926 06:15:26.514274 9936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:26.514057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:46749 (local address 127.2.110.194:53525)
0926 06:15:26.514191 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:26.514235 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:46749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":84,"threads_started":1}
I20260926 06:15:26.514473 9706 heartbeater.cc:503] Master 127.2.110.253:35513 was elected leader, sending a full tablet report...
I20260926 06:15:26.514631 9934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:26.513049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:46749 (local address 127.2.110.193:35031)
0926 06:15:26.514514 (+ 1465us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:26.514596 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:46749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1439,"thread_start_us":94,"threads_started":1}
I20260926 06:15:26.514664 9936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:26.514557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:46749 (local address 127.2.110.194:55293)
0926 06:15:26.514619 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:26.514642 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:46749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 06:15:26.515026 9933 tablet_bootstrap.cc:492] T 092fd71863fb405ca247459e6c6c3c2b P 96a7adaf68cd451ba1388e6550cf780c: Bootstrap starting.
I20260926 06:15:26.515049 9934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:26.514943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:46749 (local address 127.2.110.193:42579)
0926 06:15:26.514993 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:26.515019 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:46749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 06:15:26.515071 9570 tablet.cc:2369] T 092fd71863fb405ca247459e6c6c3c2b P 8217ae7c8e30427aa59ee96d67f4bf91: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:15:26.515476 9933 tablet_bootstrap.cc:654] T 092fd71863fb405ca247459e6c6c3c2b P 96a7adaf68cd451ba1388e6550cf780c: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:26.516009 9933 tablet_bootstrap.cc:492] T 092fd71863fb405ca247459e6c6c3c2b P 96a7adaf68cd451ba1388e6550cf780c: No bootstrap required, opened a new log
I20260926 06:15:26.516055 9933 ts_tablet_manager.cc:1397] T 092fd71863fb405ca247459e6c6c3c2b P 96a7adaf68cd451ba1388e6550cf780c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:26.516153 9933 raft_consensus.cc:348] T 092fd71863fb405ca247459e6c6c3c2b P 96a7adaf68cd451ba1388e6550cf780c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8217ae7c8e30427aa59ee96d67f4bf91" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 40801 } } peers { permanent_uuid: "96a7adaf68cd451ba1388e6550cf780c" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 38407 } } peers { permanent_uuid: "2f74a0910ea94f6aa814996cc6eff5be" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45563 } }
I20260926 06:15:26.516208 9933 raft_consensus.cc:374] T 092fd71863fb405ca247459e6c6c3c2b P 96a7adaf68cd451ba1388e6550cf780c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:26.516222 9933 raft_consensus.cc:729] T 092fd71863fb405ca247459e6c6c3c2b P 96a7adaf68cd451ba1388e6550cf780c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96a7adaf68cd451ba1388e6550cf780c, State: Initialized, Role: FOLLOWER
I20260926 06:15:26.516290 9933 consensus_queue.cc:260] T 092fd71863fb405ca247459e6c6c3c2b P 96a7adaf68cd451ba1388e6550cf780c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8217ae7c8e30427aa59ee96d67f4bf91" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 40801 } } peers { permanent_uuid: "96a7adaf68cd451ba1388e6550cf780c" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 38407 } } peers { permanent_uuid: "2f74a0910ea94f6aa814996cc6eff5be" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45563 } }
I20260926 06:15:26.516484 9933 ts_tablet_manager.cc:1428] T 092fd71863fb405ca247459e6c6c3c2b P 96a7adaf68cd451ba1388e6550cf780c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 06:15:26.517148 9939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:26.516830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:46749 (local address 127.2.110.195:60493)
0926 06:15:26.517063 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:26.517107 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:46749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":83,"threads_started":1}
I20260926 06:15:26.517521 9939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:26.517448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:46749 (local address 127.2.110.195:55363)
0926 06:15:26.517482 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:26.517505 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:46749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260926 06:15:26.554522 9843 tablet.cc:2369] T 092fd71863fb405ca247459e6c6c3c2b P 96a7adaf68cd451ba1388e6550cf780c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:15:26.746400 9932 raft_consensus.cc:482] T 092fd71863fb405ca247459e6c6c3c2b P 8217ae7c8e30427aa59ee96d67f4bf91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:15:26.746515 9932 raft_consensus.cc:504] T 092fd71863fb405ca247459e6c6c3c2b P 8217ae7c8e30427aa59ee96d67f4bf91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8217ae7c8e30427aa59ee96d67f4bf91" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 40801 } } peers { permanent_uuid: "96a7adaf68cd451ba1388e6550cf780c" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 38407 } } peers { permanent_uuid: "2f74a0910ea94f6aa814996cc6eff5be" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45563 } }
I20260926 06:15:26.746702 9932 leader_election.cc:288] T 092fd71863fb405ca247459e6c6c3c2b P 8217ae7c8e30427aa59ee96d67f4bf91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96a7adaf68cd451ba1388e6550cf780c (127.2.110.195:38407), 2f74a0910ea94f6aa814996cc6eff5be (127.2.110.194:45563)
I20260926 06:15:26.746999 9794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "092fd71863fb405ca247459e6c6c3c2b" candidate_uuid: "8217ae7c8e30427aa59ee96d67f4bf91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96a7adaf68cd451ba1388e6550cf780c" is_pre_election: true
I20260926 06:15:26.747053 9658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "092fd71863fb405ca247459e6c6c3c2b" candidate_uuid: "8217ae7c8e30427aa59ee96d67f4bf91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f74a0910ea94f6aa814996cc6eff5be" is_pre_election: true
I20260926 06:15:26.747131 9794 raft_consensus.cc:2454] T 092fd71863fb405ca247459e6c6c3c2b P 96a7adaf68cd451ba1388e6550cf780c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8217ae7c8e30427aa59ee96d67f4bf91 in term 0.
I20260926 06:15:26.747129 9658 raft_consensus.cc:2454] T 092fd71863fb405ca247459e6c6c3c2b P 2f74a0910ea94f6aa814996cc6eff5be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8217ae7c8e30427aa59ee96d67f4bf91 in term 0.
I20260926 06:15:26.747309 9455 leader_election.cc:302] T 092fd71863fb405ca247459e6c6c3c2b P 8217ae7c8e30427aa59ee96d67f4bf91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f74a0910ea94f6aa814996cc6eff5be, 8217ae7c8e30427aa59ee96d67f4bf91; no voters:
I20260926 06:15:26.747445 9932 raft_consensus.cc:2781] T 092fd71863fb405ca247459e6c6c3c2b P 8217ae7c8e30427aa59ee96d67f4bf91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:15:26.747498 9932 raft_consensus.cc:482] T 092fd71863fb405ca247459e6c6c3c2b P 8217ae7c8e30427aa59ee96d67f4bf91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:15:26.747540 9932 raft_consensus.cc:3037] T 092fd71863fb405ca247459e6c6c3c2b P 8217ae7c8e30427aa59ee96d67f4bf91 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:26.748306 9932 raft_consensus.cc:504] T 092fd71863fb405ca247459e6c6c3c2b P 8217ae7c8e30427aa59ee96d67f4bf91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8217ae7c8e30427aa59ee96d67f4bf91" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 40801 } } peers { permanent_uuid: "96a7adaf68cd451ba1388e6550cf780c" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 38407 } } peers { permanent_uuid: "2f74a0910ea94f6aa814996cc6eff5be" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45563 } }
I20260926 06:15:26.748418 9932 leader_election.cc:288] T 092fd71863fb405ca247459e6c6c3c2b P 8217ae7c8e30427aa59ee96d67f4bf91 [CANDIDATE]: Term 1 election: Requested vote from peers 96a7adaf68cd451ba1388e6550cf780c (127.2.110.195:38407), 2f74a0910ea94f6aa814996cc6eff5be (127.2.110.194:45563)
I20260926 06:15:26.748687 9658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "092fd71863fb405ca247459e6c6c3c2b" candidate_uuid: "8217ae7c8e30427aa59ee96d67f4bf91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f74a0910ea94f6aa814996cc6eff5be"
I20260926 06:15:26.748732 9794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "092fd71863fb405ca247459e6c6c3c2b" candidate_uuid: "8217ae7c8e30427aa59ee96d67f4bf91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96a7adaf68cd451ba1388e6550cf780c"
I20260926 06:15:26.748793 9794 raft_consensus.cc:3037] T 092fd71863fb405ca247459e6c6c3c2b P 96a7adaf68cd451ba1388e6550cf780c [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:26.748785 9658 raft_consensus.cc:3037] T 092fd71863fb405ca247459e6c6c3c2b P 2f74a0910ea94f6aa814996cc6eff5be [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:26.749589 9658 raft_consensus.cc:2454] T 092fd71863fb405ca247459e6c6c3c2b P 2f74a0910ea94f6aa814996cc6eff5be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8217ae7c8e30427aa59ee96d67f4bf91 in term 1.
I20260926 06:15:26.749589 9794 raft_consensus.cc:2454] T 092fd71863fb405ca247459e6c6c3c2b P 96a7adaf68cd451ba1388e6550cf780c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8217ae7c8e30427aa59ee96d67f4bf91 in term 1.
I20260926 06:15:26.749768 9455 leader_election.cc:302] T 092fd71863fb405ca247459e6c6c3c2b P 8217ae7c8e30427aa59ee96d67f4bf91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f74a0910ea94f6aa814996cc6eff5be, 8217ae7c8e30427aa59ee96d67f4bf91; no voters:
I20260926 06:15:26.749903 9932 raft_consensus.cc:2781] T 092fd71863fb405ca247459e6c6c3c2b P 8217ae7c8e30427aa59ee96d67f4bf91 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:15:26.750084 9932 raft_consensus.cc:686] T 092fd71863fb405ca247459e6c6c3c2b P 8217ae7c8e30427aa59ee96d67f4bf91 [term 1 LEADER]: Becoming Leader. State: Replica: 8217ae7c8e30427aa59ee96d67f4bf91, State: Running, Role: LEADER
I20260926 06:15:26.750135 9932 consensus_queue.cc:237] T 092fd71863fb405ca247459e6c6c3c2b P 8217ae7c8e30427aa59ee96d67f4bf91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8217ae7c8e30427aa59ee96d67f4bf91" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 40801 } } peers { permanent_uuid: "96a7adaf68cd451ba1388e6550cf780c" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 38407 } } peers { permanent_uuid: "2f74a0910ea94f6aa814996cc6eff5be" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45563 } }
I20260926 06:15:26.750466 9934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:26.750338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:46749 (local address 127.2.110.193:59111)
0926 06:15:26.750400 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:26.750433 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:46749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 06:15:26.750744 9195 catalog_manager.cc:5495] T 092fd71863fb405ca247459e6c6c3c2b P 8217ae7c8e30427aa59ee96d67f4bf91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8217ae7c8e30427aa59ee96d67f4bf91 (127.2.110.193). New cstate: current_term: 1 leader_uuid: "8217ae7c8e30427aa59ee96d67f4bf91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8217ae7c8e30427aa59ee96d67f4bf91" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 40801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96a7adaf68cd451ba1388e6550cf780c" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 38407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f74a0910ea94f6aa814996cc6eff5be" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45563 } health_report { overall_health: UNKNOWN } } }
I20260926 06:15:26.794062 9924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:26.791714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50486 (local address 127.2.110.193:40801)
0926 06:15:26.791792 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:26.791794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:26.791803 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:15:26.791829 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:26.791829 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:26.791841 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:26.791913 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:15:26.792032 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:26.792531 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:26.793306 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:26.793555 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:26.793637 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:26.793778 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:15:26.793818 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:15:26.793881 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:26.793947 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 06:15:26.793972 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 06:15:26.794991 9658 raft_consensus.cc:1261] T 092fd71863fb405ca247459e6c6c3c2b P 2f74a0910ea94f6aa814996cc6eff5be [term 1 FOLLOWER]: Refusing update from remote peer 8217ae7c8e30427aa59ee96d67f4bf91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:15:26.795001 9794 raft_consensus.cc:1261] T 092fd71863fb405ca247459e6c6c3c2b P 96a7adaf68cd451ba1388e6550cf780c [term 1 FOLLOWER]: Refusing update from remote peer 8217ae7c8e30427aa59ee96d67f4bf91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:15:26.795274 9940 consensus_queue.cc:1035] T 092fd71863fb405ca247459e6c6c3c2b P 8217ae7c8e30427aa59ee96d67f4bf91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f74a0910ea94f6aa814996cc6eff5be" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 45563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:15:26.795395 9932 consensus_queue.cc:1035] T 092fd71863fb405ca247459e6c6c3c2b P 8217ae7c8e30427aa59ee96d67f4bf91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96a7adaf68cd451ba1388e6550cf780c" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 38407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:15:26.796018 9842 heartbeater.cc:503] Master 127.2.110.253:35513 was elected leader, sending a full tablet report...
I20260926 06:15:26.796087 9939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:26.795944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:46749 (local address 127.2.110.195:38727)
0926 06:15:26.796002 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:26.796047 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:46749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 06:15:26.796396 9936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:26.796267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:46749 (local address 127.2.110.194:51425)
0926 06:15:26.796334 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:26.796359 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:46749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 06:15:26.798084 9936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:26.797768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:46749 (local address 127.2.110.194:35755)
0926 06:15:26.797923 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:26.797969 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:46749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260926 06:15:26.798147 9939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:26.797954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:46749 (local address 127.2.110.195:60255)
0926 06:15:26.798078 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:26.798117 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:46749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260926 06:15:26.799108 9934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:26.798968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:46749 (local address 127.2.110.193:43631)
0926 06:15:26.799045 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:26.799075 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:46749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 06:15:26.800055 9195 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:15:26.800140 9195 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:15:26.802126 9195 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4a5a9e72e816463b9cfcb45b2f858088: Sending DeleteTablet for 3 replicas of tablet 092fd71863fb405ca247459e6c6c3c2b
I20260926 06:15:26.802582 9638 tablet_service.cc:1514] Processing DeleteTablet for tablet 092fd71863fb405ca247459e6c6c3c2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:15:26 UTC) from {username='oryx', principal='oryx/127.2.110.253@KRBTEST.COM'} at 127.0.0.1:45958
I20260926 06:15:26.802616 9773 tablet_service.cc:1514] Processing DeleteTablet for tablet 092fd71863fb405ca247459e6c6c3c2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:15:26 UTC) from {username='oryx', principal='oryx/127.2.110.253@KRBTEST.COM'} at 127.0.0.1:60656
I20260926 06:15:26.802897 9501 tablet_service.cc:1514] Processing DeleteTablet for tablet 092fd71863fb405ca247459e6c6c3c2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:15:26 UTC) from {username='oryx', principal='oryx/127.2.110.253@KRBTEST.COM'} at 127.0.0.1:50484
I20260926 06:15:26.802970 9952 tablet_replica.cc:330] T 092fd71863fb405ca247459e6c6c3c2b P 2f74a0910ea94f6aa814996cc6eff5be: stopping tablet replica
I20260926 06:15:26.803040 9952 raft_consensus.cc:2229] T 092fd71863fb405ca247459e6c6c3c2b P 2f74a0910ea94f6aa814996cc6eff5be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:15:26.803102 9952 raft_consensus.cc:2258] T 092fd71863fb405ca247459e6c6c3c2b P 2f74a0910ea94f6aa814996cc6eff5be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:15:26.803092 9953 tablet_replica.cc:330] T 092fd71863fb405ca247459e6c6c3c2b P 96a7adaf68cd451ba1388e6550cf780c: stopping tablet replica
I20260926 06:15:26.803143 9953 raft_consensus.cc:2229] T 092fd71863fb405ca247459e6c6c3c2b P 96a7adaf68cd451ba1388e6550cf780c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:15:26.803196 9953 raft_consensus.cc:2258] T 092fd71863fb405ca247459e6c6c3c2b P 96a7adaf68cd451ba1388e6550cf780c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:15:26.803402 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 9343
I20260926 06:15:26.803529 9953 ts_tablet_manager.cc:1905] T 092fd71863fb405ca247459e6c6c3c2b P 96a7adaf68cd451ba1388e6550cf780c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:15:26.803833 9952 ts_tablet_manager.cc:1905] T 092fd71863fb405ca247459e6c6c3c2b P 2f74a0910ea94f6aa814996cc6eff5be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:15:26.805243 9953 ts_tablet_manager.cc:1918] T 092fd71863fb405ca247459e6c6c3c2b P 96a7adaf68cd451ba1388e6550cf780c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:15:26.805303 9953 log.cc:1196] T 092fd71863fb405ca247459e6c6c3c2b P 96a7adaf68cd451ba1388e6550cf780c: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-2/wal/wals/092fd71863fb405ca247459e6c6c3c2b
I20260926 06:15:26.805572 9953 ts_tablet_manager.cc:1939] T 092fd71863fb405ca247459e6c6c3c2b P 96a7adaf68cd451ba1388e6550cf780c: Deleting consensus metadata
I20260926 06:15:26.805949 9181 catalog_manager.cc:4841] TS 96a7adaf68cd451ba1388e6550cf780c (127.2.110.195:38407): tablet 092fd71863fb405ca247459e6c6c3c2b (table test-table [id=0d1cc70a5a854a4b9e20f9a92624c7d9]) successfully deleted
I20260926 06:15:26.806450 9952 ts_tablet_manager.cc:1918] T 092fd71863fb405ca247459e6c6c3c2b P 2f74a0910ea94f6aa814996cc6eff5be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:15:26.806561 9952 log.cc:1196] T 092fd71863fb405ca247459e6c6c3c2b P 2f74a0910ea94f6aa814996cc6eff5be: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403292369990-2491-0/minicluster-data/ts-1/wal/wals/092fd71863fb405ca247459e6c6c3c2b
I20260926 06:15:26.806948 9952 ts_tablet_manager.cc:1939] T 092fd71863fb405ca247459e6c6c3c2b P 2f74a0910ea94f6aa814996cc6eff5be: Deleting consensus metadata
I20260926 06:15:26.807379 9181 catalog_manager.cc:4841] TS 2f74a0910ea94f6aa814996cc6eff5be (127.2.110.194:45563): tablet 092fd71863fb405ca247459e6c6c3c2b (table test-table [id=0d1cc70a5a854a4b9e20f9a92624c7d9]) successfully deleted
I20260926 06:15:26.811860 9925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:26.811707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.193:40801 (local address 127.0.0.1:50488)
0926 06:15:26.811777 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:26.811816 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.193:40801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 06:15:26.811887 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 9575
W20260926 06:15:26.812073 9181 catalog_manager.cc:4584] TS 8217ae7c8e30427aa59ee96d67f4bf91 (127.2.110.193:40801): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 092fd71863fb405ca247459e6c6c3c2b: Network error: Client connection negotiation failed: client connection to 127.2.110.193:40801: connect: Connection refused (error 111)
I20260926 06:15:26.818027 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 9711
I20260926 06:15:26.823310 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 9062
I20260926 06:15:26.828812 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 9250
2026-09-26T06:15:26Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4940 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:15:26 dist-test-slave-hndd krb5kdc[9957](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:15:26 dist-test-slave-hndd krb5kdc[9957](info): set up 2 sockets
Sep 26 06:15:26 dist-test-slave-hndd krb5kdc[9957](info): commencing operation
krb5kdc: starting...
W20260926 06:15:27.902040 2491 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Sep 26 06:15:27 dist-test-slave-hndd krb5kdc[9957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:15:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:15:28Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/krb5kdc/oryx_127.2.110.254.keytab.
Entry for principal oryx/127.2.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/krb5kdc/oryx_127.2.110.254.keytab.
I20260926 06:15:28.033773 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:46239
--webserver_interface=127.2.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/krb5kdc/oryx_127.2.110.254.keytab
--principal=oryx/127.2.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:40309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.110.254:46239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:15:28.115552 9970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:28.115832 9970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:28.115859 9970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:28.117053 9970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:15:28.117097 9970 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 06:15:28.117111 9970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:28.117125 9970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:28.117138 9970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:15:28.117151 9970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:15:28.118726 9970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:40309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.110.254:46239
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/krb5kdc/oryx_127.2.110.254.keytab
--principal=oryx/127.2.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:46239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:28.118950 9970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:28.119133 9970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:28.124360 9975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:28.124395 9978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:28.124363 9976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:28.124543 9970 server_base.cc:1034] running on GCE node
I20260926 06:15:28.125046 9970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:28.125420 9970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:28.126638 9970 hybrid_clock.cc:648] HybridClock initialized: now 1790403328126604 us; error 40 us; skew 500 ppm
Sep 26 06:15:28 dist-test-slave-hndd krb5kdc[9957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403328, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:28.130012 9970 init.cc:377] Logged in from keytab as oryx/127.2.110.254@KRBTEST.COM (short username oryx)
I20260926 06:15:28.130604 9970 webserver.cc:460] Webserver started at http://127.2.110.254:39781/ using document root <none> and password file <none>
I20260926 06:15:28.130900 9970 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:28.130982 9970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:28.131150 9970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:28.132354 9970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/master-0/data/instance:
uuid: "24536bc278a44d40bfcee7eb68808bf6"
format_stamp: "Formatted at 2026-09-26 06:15:28 on dist-test-slave-hndd"
server_key: "bb51a40f61febe9475762c61ba82496f"
server_key_iv: "a67dc0b1afee77a9fe27c3d4821fb6df"
server_key_version: "encryptionkey@0"
I20260926 06:15:28.132767 9970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/master-0/wal/instance:
uuid: "24536bc278a44d40bfcee7eb68808bf6"
format_stamp: "Formatted at 2026-09-26 06:15:28 on dist-test-slave-hndd"
server_key: "bb51a40f61febe9475762c61ba82496f"
server_key_iv: "a67dc0b1afee77a9fe27c3d4821fb6df"
server_key_version: "encryptionkey@0"
I20260926 06:15:28.134959 9970 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260926 06:15:28.136125 9985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:28.136493 9970 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:28.136674 9970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "24536bc278a44d40bfcee7eb68808bf6"
format_stamp: "Formatted at 2026-09-26 06:15:28 on dist-test-slave-hndd"
server_key: "bb51a40f61febe9475762c61ba82496f"
server_key_iv: "a67dc0b1afee77a9fe27c3d4821fb6df"
server_key_version: "encryptionkey@0"
I20260926 06:15:28.136807 9970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:28.165602 9970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:28.168395 9970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:28.168654 9970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:28.174153 9970 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.254:46239
I20260926 06:15:28.174211 10047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.254:46239 every 8 connection(s)
I20260926 06:15:28.174574 9970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:15:28.175314 10048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:28.178464 10048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24536bc278a44d40bfcee7eb68808bf6: Bootstrap starting.
I20260926 06:15:28.179390 10048 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24536bc278a44d40bfcee7eb68808bf6: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:28.179781 10048 log.cc:824] T 00000000000000000000000000000000 P 24536bc278a44d40bfcee7eb68808bf6: Log is configured to *not* fsync() on all Append() calls
I20260926 06:15:28.180518 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 9970
I20260926 06:15:28.180747 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/master-0/wal/instance
I20260926 06:15:28.181123 2491 external_mini_cluster.cc:1428] Setting key 917b8e254bd494be5f5c064b90a86345
I20260926 06:15:28.181203 10048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24536bc278a44d40bfcee7eb68808bf6: No bootstrap required, opened a new log
I20260926 06:15:28.184301 10048 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24536bc278a44d40bfcee7eb68808bf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24536bc278a44d40bfcee7eb68808bf6" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46239 } }
I20260926 06:15:28.184540 10048 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24536bc278a44d40bfcee7eb68808bf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:28.184583 10048 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24536bc278a44d40bfcee7eb68808bf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24536bc278a44d40bfcee7eb68808bf6, State: Initialized, Role: FOLLOWER
I20260926 06:15:28.184785 10048 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24536bc278a44d40bfcee7eb68808bf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24536bc278a44d40bfcee7eb68808bf6" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46239 } }
I20260926 06:15:28.184888 10048 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24536bc278a44d40bfcee7eb68808bf6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:15:28.184926 10048 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24536bc278a44d40bfcee7eb68808bf6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:15:28.184973 10048 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24536bc278a44d40bfcee7eb68808bf6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:28.185775 10048 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24536bc278a44d40bfcee7eb68808bf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24536bc278a44d40bfcee7eb68808bf6" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46239 } }
I20260926 06:15:28.185870 10048 leader_election.cc:302] T 00000000000000000000000000000000 P 24536bc278a44d40bfcee7eb68808bf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24536bc278a44d40bfcee7eb68808bf6; no voters:
I20260926 06:15:28.186094 10048 leader_election.cc:288] T 00000000000000000000000000000000 P 24536bc278a44d40bfcee7eb68808bf6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:15:28.186129 10053 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24536bc278a44d40bfcee7eb68808bf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:15:28.186242 10053 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24536bc278a44d40bfcee7eb68808bf6 [term 1 LEADER]: Becoming Leader. State: Replica: 24536bc278a44d40bfcee7eb68808bf6, State: Running, Role: LEADER
I20260926 06:15:28.186425 10048 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24536bc278a44d40bfcee7eb68808bf6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:15:28.186385 10053 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24536bc278a44d40bfcee7eb68808bf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24536bc278a44d40bfcee7eb68808bf6" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46239 } }
I20260926 06:15:28.186906 10054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24536bc278a44d40bfcee7eb68808bf6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24536bc278a44d40bfcee7eb68808bf6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24536bc278a44d40bfcee7eb68808bf6" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46239 } } }
I20260926 06:15:28.187274 10054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24536bc278a44d40bfcee7eb68808bf6 [sys.catalog]: This master's current role is: LEADER
I20260926 06:15:28.187201 10055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24536bc278a44d40bfcee7eb68808bf6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24536bc278a44d40bfcee7eb68808bf6. Latest consensus state: current_term: 1 leader_uuid: "24536bc278a44d40bfcee7eb68808bf6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24536bc278a44d40bfcee7eb68808bf6" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 46239 } } }
I20260926 06:15:28.187682 10055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24536bc278a44d40bfcee7eb68808bf6 [sys.catalog]: This master's current role is: LEADER
W20260926 06:15:28.188199 10069 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 24536bc278a44d40bfcee7eb68808bf6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 06:15:28.188261 10069 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 06:15:28.188323 10062 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:15:28.188491 10062 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:15:28.190493 10062 catalog_manager.cc:1347] Generated new cluster ID: 52b8a3159ef04045a700cad9f84c6879
I20260926 06:15:28.190580 10062 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 06:15:28 dist-test-slave-hndd krb5kdc[9957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM
I20260926 06:15:28.198601 10052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:28.181903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53634 (local address 127.2.110.254:46239)
0926 06:15:28.182111 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:28.182121 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:28.182154 (+ 33us) server_negotiation.cc:407] Connection header received
0926 06:15:28.184269 (+ 2115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:28.184280 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:28.184355 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:28.184510 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:28.185422 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.186606 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.187683 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.188309 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.188433 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:28.193184 (+ 4751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:28.193191 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:28.193192 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:28.193210 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:28.196433 (+ 3223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:28.196918 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:28.196921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:28.196922 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:28.197028 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:28.197202 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:28.197203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:28.197203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:28.197531 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:28.197680 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:28.198198 (+ 518us) server_negotiation.cc:299] Negotiation successful
0926 06:15:28.198342 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":109,"threads_started":1}
I20260926 06:15:28.222635 10062 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:15:28.223511 10062 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:15:28.225803 10062 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24536bc278a44d40bfcee7eb68808bf6: Generated new TSK 0
I20260926 06:15:28.226128 10062 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/krb5kdc/oryx_127.2.110.193.keytab.
Entry for principal oryx/127.2.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/krb5kdc/oryx_127.2.110.193.keytab.
I20260926 06:15:28.259168 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.193:0
--local_ip_for_outbound_sockets=127.2.110.193
--webserver_interface=127.2.110.193
--webserver_port=0
--tserver_master_addrs=127.2.110.254:46239
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/krb5kdc/oryx_127.2.110.193.keytab
--principal=oryx/127.2.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:40309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:15:28.345857 10074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:28.346086 10074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:28.346112 10074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:28.347285 10074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:28.347340 10074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:28.347362 10074 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 06:15:28.347381 10074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.193
I20260926 06:15:28.348893 10074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:40309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/krb5kdc/oryx_127.2.110.193.keytab
--principal=oryx/127.2.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.110.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.110.254:46239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.193
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:28.349098 10074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:28.349287 10074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:28.355412 10079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:28.355543 10082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:28.355412 10080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:28.355837 10074 server_base.cc:1034] running on GCE node
I20260926 06:15:28.356081 10074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:28.356309 10074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:28.357488 10074 hybrid_clock.cc:648] HybridClock initialized: now 1790403328357488 us; error 42 us; skew 500 ppm
Sep 26 06:15:28 dist-test-slave-hndd krb5kdc[9957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403328, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:28.360471 10074 init.cc:377] Logged in from keytab as oryx/127.2.110.193@KRBTEST.COM (short username oryx)
I20260926 06:15:28.361078 10074 webserver.cc:460] Webserver started at http://127.2.110.193:41161/ using document root <none> and password file <none>
I20260926 06:15:28.361330 10074 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:28.361465 10074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:28.361609 10074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:28.362703 10074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-0/data/instance:
uuid: "56036654e641486b99a44119fcff490f"
format_stamp: "Formatted at 2026-09-26 06:15:28 on dist-test-slave-hndd"
server_key: "60b456a7e6c81b07acbcef2d395ca4be"
server_key_iv: "0c8eff8b3d847a56a04a6f729826a3e0"
server_key_version: "encryptionkey@0"
I20260926 06:15:28.363087 10074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-0/wal/instance:
uuid: "56036654e641486b99a44119fcff490f"
format_stamp: "Formatted at 2026-09-26 06:15:28 on dist-test-slave-hndd"
server_key: "60b456a7e6c81b07acbcef2d395ca4be"
server_key_iv: "0c8eff8b3d847a56a04a6f729826a3e0"
server_key_version: "encryptionkey@0"
I20260926 06:15:28.364658 10074 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:15:28.365613 10089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:28.365994 10074 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:15:28.366253 10074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-0/wal
uuid: "56036654e641486b99a44119fcff490f"
format_stamp: "Formatted at 2026-09-26 06:15:28 on dist-test-slave-hndd"
server_key: "60b456a7e6c81b07acbcef2d395ca4be"
server_key_iv: "0c8eff8b3d847a56a04a6f729826a3e0"
server_key_version: "encryptionkey@0"
I20260926 06:15:28.366382 10074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:28.392979 10074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:28.395676 10074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:28.395912 10074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:28.396586 10074 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:15:28.396653 10074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:28.396706 10074 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:15:28.396730 10074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 06:15:28 dist-test-slave-hndd krb5kdc[9957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403328, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.110.193@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM
I20260926 06:15:28.405539 10074 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.193:37843
I20260926 06:15:28.405612 10204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.193:37843 every 8 connection(s)
I20260926 06:15:28.405983 10074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
I20260926 06:15:28.407151 10052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:28.396988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:35393 (local address 127.2.110.254:46239)
0926 06:15:28.397047 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:28.397050 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:28.397303 (+ 253us) server_negotiation.cc:407] Connection header received
0926 06:15:28.397869 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:28.397871 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:28.397889 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:28.397973 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:28.398712 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.399223 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.400268 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.400494 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.400577 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:28.403622 (+ 3045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:28.403625 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:28.403625 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:28.403634 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:28.405612 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:28.406148 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:28.406157 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:28.406158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:28.406228 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:28.406569 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:28.406570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:28.406570 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:28.406697 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:28.406754 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:28.407009 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 06:15:28.407087 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 06:15:28.407325 10098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:28.396995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:46239 (local address 127.2.110.193:35393)
0926 06:15:28.397149 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:28.397190 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:28.397747 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:28.398054 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:28.398058 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:28.398367 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:28.398604 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:28.398610 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.399331 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.399332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:28.400166 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:28.400168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.400584 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.400585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:28.400747 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:28.401462 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:28.401472 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:28.403502 (+ 2030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:28.405720 (+ 2218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:28.405725 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:28.405727 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:28.406050 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:28.406310 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:28.406311 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:28.406312 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:28.406495 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:28.406761 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:28.406764 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:28.406892 (+ 128us) client_negotiation.cc:769] Sending connection context
0926 06:15:28.406979 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 06:15:28.407114 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
I20260926 06:15:28.407833 10205 heartbeater.cc:347] Connected to a master server at 127.2.110.254:46239
I20260926 06:15:28.407922 10205 heartbeater.cc:464] Registering TS with master...
I20260926 06:15:28.408113 10205 heartbeater.cc:511] Master 127.2.110.254:46239 requested a full tablet report, sending...
I20260926 06:15:28.408902 10001 ts_manager.cc:194] Registered new tserver with Master: 56036654e641486b99a44119fcff490f (127.2.110.193:37843)
I20260926 06:15:28.410192 10001 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.110.193@KRBTEST.COM'} at 127.2.110.193:35393
I20260926 06:15:28.415603 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 10074
I20260926 06:15:28.415757 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-0/wal/instance
I20260926 06:15:28.416019 2491 external_mini_cluster.cc:1428] Setting key 4a9e7c8dcce2312d8696c50713768e94
I20260926 06:15:28.416905 10211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:28.408959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:46239 (local address 127.2.110.193:43001)
0926 06:15:28.409085 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:28.409096 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:28.409165 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:28.409534 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:28.409536 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:28.409809 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:28.409898 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:28.409900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.411209 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.411210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:28.411804 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:28.411806 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.412171 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.412172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:28.412231 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:28.412880 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:28.412884 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:28.413211 (+ 327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:28.415359 (+ 2148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:28.415362 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:28.415362 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:28.415687 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:28.416009 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:28.416011 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:28.416011 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:28.416093 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:28.416530 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:28.416532 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:28.416605 (+ 73us) client_negotiation.cc:769] Sending connection context
0926 06:15:28.416687 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 06:15:28.416825 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":74,"threads_started":1}
I20260926 06:15:28.416917 10052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:28.409044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:43001 (local address 127.2.110.254:46239)
0926 06:15:28.409106 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:28.409109 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:28.409158 (+ 49us) server_negotiation.cc:407] Connection header received
0926 06:15:28.409331 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:28.409362 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:28.409377 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:28.409455 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:28.410135 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.411065 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.411891 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.412078 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.412139 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:28.413304 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:28.413305 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:28.413306 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:28.413312 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:28.415182 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:28.415827 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:28.415828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:28.415828 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:28.415905 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:28.416237 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:28.416238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:28.416238 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:28.416411 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:28.416479 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:28.416734 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 06:15:28.416829 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
WARNING: no policy specified for oryx/127.2.110.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.110.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/krb5kdc/oryx_127.2.110.194.keytab.
Entry for principal oryx/127.2.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/krb5kdc/oryx_127.2.110.194.keytab.
I20260926 06:15:28.444308 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.194:0
--local_ip_for_outbound_sockets=127.2.110.194
--webserver_interface=127.2.110.194
--webserver_port=0
--tserver_master_addrs=127.2.110.254:46239
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/krb5kdc/oryx_127.2.110.194.keytab
--principal=oryx/127.2.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:40309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:15:28.526018 10214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:28.526269 10214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:28.526312 10214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:28.527551 10214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:28.527626 10214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:28.527664 10214 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 06:15:28.527719 10214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.194
I20260926 06:15:28.529299 10214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:40309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/krb5kdc/oryx_127.2.110.194.keytab
--principal=oryx/127.2.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.110.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.110.254:46239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.194
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:28.529641 10214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:28.529883 10214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:28.535195 10219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:28.535208 10222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:28.535208 10220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:28.535269 10214 server_base.cc:1034] running on GCE node
I20260926 06:15:28.535678 10214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:28.535977 10214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:28.537175 10214 hybrid_clock.cc:648] HybridClock initialized: now 1790403328537160 us; error 28 us; skew 500 ppm
Sep 26 06:15:28 dist-test-slave-hndd krb5kdc[9957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403328, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:28.539925 10214 init.cc:377] Logged in from keytab as oryx/127.2.110.194@KRBTEST.COM (short username oryx)
I20260926 06:15:28.540474 10214 webserver.cc:460] Webserver started at http://127.2.110.194:39289/ using document root <none> and password file <none>
I20260926 06:15:28.540729 10214 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:28.540805 10214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:28.540979 10214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:28.542143 10214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-1/data/instance:
uuid: "b54d024ec6964d07844331e5a81db31e"
format_stamp: "Formatted at 2026-09-26 06:15:28 on dist-test-slave-hndd"
server_key: "7ce5890a802094ae6675b21ed35a8a7c"
server_key_iv: "d8c9a1467370fea817733253d83c284d"
server_key_version: "encryptionkey@0"
I20260926 06:15:28.542474 10214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-1/wal/instance:
uuid: "b54d024ec6964d07844331e5a81db31e"
format_stamp: "Formatted at 2026-09-26 06:15:28 on dist-test-slave-hndd"
server_key: "7ce5890a802094ae6675b21ed35a8a7c"
server_key_iv: "d8c9a1467370fea817733253d83c284d"
server_key_version: "encryptionkey@0"
I20260926 06:15:28.543931 10214 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 06:15:28.544696 10229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:28.544898 10214 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:15:28.545010 10214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-1/wal
uuid: "b54d024ec6964d07844331e5a81db31e"
format_stamp: "Formatted at 2026-09-26 06:15:28 on dist-test-slave-hndd"
server_key: "7ce5890a802094ae6675b21ed35a8a7c"
server_key_iv: "d8c9a1467370fea817733253d83c284d"
server_key_version: "encryptionkey@0"
I20260926 06:15:28.545090 10214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:28.558388 10214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:28.560702 10214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:28.560905 10214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:28.561589 10214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:15:28.561653 10214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:28.561698 10214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:15:28.561769 10214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 06:15:28 dist-test-slave-hndd krb5kdc[9957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403328, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.110.194@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM
I20260926 06:15:28.569286 10214 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.194:36115
I20260926 06:15:28.569419 10344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.194:36115 every 8 connection(s)
I20260926 06:15:28.569837 10214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
I20260926 06:15:28.571551 10052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:28.562016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:34253 (local address 127.2.110.254:46239)
0926 06:15:28.562086 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:28.562088 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:28.562313 (+ 225us) server_negotiation.cc:407] Connection header received
0926 06:15:28.562865 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:28.562866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:28.562879 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:28.562942 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:28.563641 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.564146 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.565262 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.565535 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.565622 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:28.568666 (+ 3044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:28.568668 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:28.568668 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:28.568677 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:28.570460 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:28.570836 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:28.570837 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:28.570837 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:28.570891 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:28.571070 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:28.571071 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:28.571071 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:28.571183 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:28.571233 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:28.571428 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 06:15:28.571489 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 06:15:28.571834 10238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:28.562009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:46239 (local address 127.2.110.194:34253)
0926 06:15:28.562185 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:28.562220 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:28.562750 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:28.563011 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:28.563015 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:28.563281 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:28.563537 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:28.563542 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.564247 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.564249 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:28.565160 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:28.565163 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.565634 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.565635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:28.565758 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:28.566348 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:28.566365 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:28.568537 (+ 2172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:28.570524 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:28.570527 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:28.570529 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:28.570758 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:28.570937 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:28.570937 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:28.570937 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:28.571020 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:28.571222 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:28.571224 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:28.571340 (+ 116us) client_negotiation.cc:769] Sending connection context
0926 06:15:28.571419 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 06:15:28.571555 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":90,"threads_started":1}
I20260926 06:15:28.572368 10345 heartbeater.cc:347] Connected to a master server at 127.2.110.254:46239
I20260926 06:15:28.572454 10345 heartbeater.cc:464] Registering TS with master...
I20260926 06:15:28.572642 10345 heartbeater.cc:511] Master 127.2.110.254:46239 requested a full tablet report, sending...
I20260926 06:15:28.573092 10001 ts_manager.cc:194] Registered new tserver with Master: b54d024ec6964d07844331e5a81db31e (127.2.110.194:36115)
I20260926 06:15:28.573684 10001 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.110.194@KRBTEST.COM'} at 127.2.110.194:34253
I20260926 06:15:28.579792 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 10214
I20260926 06:15:28.579931 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-1/wal/instance
I20260926 06:15:28.580200 2491 external_mini_cluster.cc:1428] Setting key 56cfa320aa0abe844c5f9834f970a056
I20260926 06:15:28.580518 10351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:28.573476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:46239 (local address 127.2.110.194:60009)
0926 06:15:28.573890 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:28.573912 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:28.574080 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:28.574215 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:28.574216 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:28.574464 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:28.574528 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:28.574530 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.575357 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.575358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:28.576097 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:28.576100 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.576617 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.576618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:28.576686 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:28.577289 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:28.577295 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:28.577697 (+ 402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:28.579113 (+ 1416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:28.579114 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:28.579114 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:28.579314 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:28.579639 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:28.579640 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:28.579640 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:28.579721 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:28.580150 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:28.580151 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:28.580210 (+ 59us) client_negotiation.cc:769] Sending connection context
0926 06:15:28.580293 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 06:15:28.580422 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":133,"threads_started":1}
I20260926 06:15:28.580572 10052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:28.573535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:60009 (local address 127.2.110.254:46239)
0926 06:15:28.573583 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:28.573585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:28.574078 (+ 493us) server_negotiation.cc:407] Connection header received
0926 06:15:28.574152 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:28.574152 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:28.574161 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:28.574206 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:28.574687 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.575221 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.576198 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.576407 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.576472 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:28.577781 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:28.577782 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:28.577782 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:28.577788 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:28.579011 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:28.579407 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:28.579408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:28.579408 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:28.579468 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:28.579803 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:28.579803 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:28.579804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:28.580078 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:28.580138 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:28.580342 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 06:15:28.580459 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
WARNING: no policy specified for oryx/127.2.110.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.110.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/krb5kdc/oryx_127.2.110.195.keytab.
Entry for principal oryx/127.2.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/krb5kdc/oryx_127.2.110.195.keytab.
I20260926 06:15:28.609968 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.195:0
--local_ip_for_outbound_sockets=127.2.110.195
--webserver_interface=127.2.110.195
--webserver_port=0
--tserver_master_addrs=127.2.110.254:46239
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/krb5kdc/oryx_127.2.110.195.keytab
--principal=oryx/127.2.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:40309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:15:28.702451 10354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:28.702682 10354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:28.702708 10354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:28.703830 10354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:28.703867 10354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:28.703889 10354 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 06:15:28.703907 10354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.195
I20260926 06:15:28.705547 10354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:40309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/krb5kdc/oryx_127.2.110.195.keytab
--principal=oryx/127.2.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.110.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.110.254:46239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.195
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:28.705761 10354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:28.705943 10354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:28.712175 10359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:28.712175 10362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:28.712218 10360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:28.712607 10354 server_base.cc:1034] running on GCE node
I20260926 06:15:28.712872 10354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:28.713106 10354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:28.714284 10354 hybrid_clock.cc:648] HybridClock initialized: now 1790403328714259 us; error 61 us; skew 500 ppm
Sep 26 06:15:28 dist-test-slave-hndd krb5kdc[9957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403328, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:28.717875 10354 init.cc:377] Logged in from keytab as oryx/127.2.110.195@KRBTEST.COM (short username oryx)
I20260926 06:15:28.718604 10354 webserver.cc:460] Webserver started at http://127.2.110.195:36863/ using document root <none> and password file <none>
I20260926 06:15:28.718902 10354 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:28.718981 10354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:28.719138 10354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:28.720255 10354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-2/data/instance:
uuid: "df439d14be854ee5a671c9817319a4d4"
format_stamp: "Formatted at 2026-09-26 06:15:28 on dist-test-slave-hndd"
server_key: "f6e2f04c2dbfa498a1b57f5b90f3ea06"
server_key_iv: "97112edbed398cdb447b34237d056f87"
server_key_version: "encryptionkey@0"
I20260926 06:15:28.720624 10354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-2/wal/instance:
uuid: "df439d14be854ee5a671c9817319a4d4"
format_stamp: "Formatted at 2026-09-26 06:15:28 on dist-test-slave-hndd"
server_key: "f6e2f04c2dbfa498a1b57f5b90f3ea06"
server_key_iv: "97112edbed398cdb447b34237d056f87"
server_key_version: "encryptionkey@0"
I20260926 06:15:28.722134 10354 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:15:28.723035 10369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:28.723210 10354 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 06:15:28.723312 10354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-2/wal
uuid: "df439d14be854ee5a671c9817319a4d4"
format_stamp: "Formatted at 2026-09-26 06:15:28 on dist-test-slave-hndd"
server_key: "f6e2f04c2dbfa498a1b57f5b90f3ea06"
server_key_iv: "97112edbed398cdb447b34237d056f87"
server_key_version: "encryptionkey@0"
I20260926 06:15:28.723400 10354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:28.739883 10354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:28.742313 10354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:28.742503 10354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:28.743311 10354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:15:28.743374 10354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:28.743435 10354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:15:28.743466 10354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 06:15:28 dist-test-slave-hndd krb5kdc[9957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403328, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.110.195@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM
I20260926 06:15:28.750820 10354 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.195:34709
I20260926 06:15:28.750891 10484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.195:34709 every 8 connection(s)
I20260926 06:15:28.751260 10354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
I20260926 06:15:28.753302 10052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:28.743761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:44469 (local address 127.2.110.254:46239)
0926 06:15:28.743837 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:28.743840 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:28.744065 (+ 225us) server_negotiation.cc:407] Connection header received
0926 06:15:28.744553 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:28.744554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:28.744567 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:28.744651 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:28.745281 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.746145 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.747209 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.747428 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.747519 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:28.750411 (+ 2892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:28.750413 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:28.750413 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:28.750422 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:28.752187 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:28.752553 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:28.752554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:28.752554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:28.752605 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:28.752777 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:28.752778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:28.752778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:28.752884 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:28.752933 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:28.753169 (+ 236us) server_negotiation.cc:299] Negotiation successful
0926 06:15:28.753230 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 06:15:28.753556 10378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:28.743760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:46239 (local address 127.2.110.195:44469)
0926 06:15:28.743931 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:28.743972 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:28.744459 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:28.744710 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:28.744713 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:28.744969 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:28.745172 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:28.745176 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.746252 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.746253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:28.747104 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:28.747106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.747531 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.747532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:28.747693 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:28.748277 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:28.748285 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:28.750299 (+ 2014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:28.752266 (+ 1967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:28.752270 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:28.752272 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:28.752484 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:28.752645 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:28.752646 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:28.752646 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:28.752727 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:28.752942 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:28.752946 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:28.753060 (+ 114us) client_negotiation.cc:769] Sending connection context
0926 06:15:28.753143 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 06:15:28.753267 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":103,"threads_started":1}
I20260926 06:15:28.754026 10485 heartbeater.cc:347] Connected to a master server at 127.2.110.254:46239
I20260926 06:15:28.754097 10485 heartbeater.cc:464] Registering TS with master...
I20260926 06:15:28.754300 10485 heartbeater.cc:511] Master 127.2.110.254:46239 requested a full tablet report, sending...
I20260926 06:15:28.754670 10002 ts_manager.cc:194] Registered new tserver with Master: df439d14be854ee5a671c9817319a4d4 (127.2.110.195:34709)
I20260926 06:15:28.755159 10002 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.110.195@KRBTEST.COM'} at 127.2.110.195:44469
I20260926 06:15:28.755533 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 10354
I20260926 06:15:28.755640 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-2/wal/instance
I20260926 06:15:28.755857 2491 external_mini_cluster.cc:1428] Setting key dcc8da6607958eb28b9f5571bad9c02c
I20260926 06:15:28.756675 2491 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 06:15:28.761703 10491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:28.755083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:46239 (local address 127.2.110.195:57965)
0926 06:15:28.755247 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:28.755257 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:28.755318 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:28.755482 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:28.755483 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:28.755682 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:28.755754 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:28.755755 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.756692 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.756692 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:28.757255 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:28.757257 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.757754 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.757755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:28.757823 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:28.758326 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:28.758332 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:28.758778 (+ 446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:28.760556 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:28.760557 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:28.760557 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:28.760728 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:28.760971 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:28.760972 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:28.760972 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:28.761049 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:28.761401 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:28.761402 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:28.761442 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 06:15:28.761498 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 06:15:28.761609 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":74,"threads_started":1}
I20260926 06:15:28.761744 10052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:28.755145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:57965 (local address 127.2.110.254:46239)
0926 06:15:28.755190 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:28.755193 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:28.755314 (+ 121us) server_negotiation.cc:407] Connection header received
0926 06:15:28.755393 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:28.755394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:28.755405 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:28.755470 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:28.755846 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.756546 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.757367 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.757671 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.757749 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:28.758878 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:28.758879 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:28.758880 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:28.758886 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:28.760470 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:28.760817 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:28.760819 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:28.760819 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:28.760881 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:28.761122 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:28.761123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:28.761123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:28.761292 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:28.761396 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:28.761549 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 06:15:28.761637 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
Sep 26 06:15:28 dist-test-slave-hndd krb5kdc[9957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403328, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 06:15:28 dist-test-slave-hndd krb5kdc[9957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:15:28 dist-test-slave-hndd krb5kdc[9957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 06:15:28.781164 10052 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:28.774668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53638 (local address 127.2.110.254:46239)
0926 06:15:28.774731 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:28.774735 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:28.774750 (+ 15us) server_negotiation.cc:407] Connection header received
0926 06:15:28.774777 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:28.774778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:28.774788 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:28.774851 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:28.775607 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.776304 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.777102 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.777305 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.777476 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:28.781056 (+ 3580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53638: BlockingRecv error: recv got EOF from 127.0.0.1:53638 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 26 06:15:28 dist-test-slave-hndd krb5kdc[9957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403328, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM
I20260926 06:15:28.789439 10052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:28.782017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53648 (local address 127.2.110.254:46239)
0926 06:15:28.782114 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:28.782117 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:28.782201 (+ 84us) server_negotiation.cc:407] Connection header received
0926 06:15:28.782300 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:28.782301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:28.782318 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:28.782379 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:28.782863 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.783363 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.784067 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.784274 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.784369 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:28.786742 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:28.786744 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:28.786744 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:28.786753 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:28.788200 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:28.788591 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:28.788592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:28.788592 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:28.788651 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:28.788862 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:28.788862 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:28.788863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:28.789000 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:28.789105 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:28.789211 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 06:15:28.789303 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 06:15:28.791474 10002 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53648:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:15:28.793046 10002 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:15:28.800014 10508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:28.795765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.195:34709 (local address 127.0.0.1:41970)
0926 06:15:28.796077 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:28.796087 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:28.796128 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:28.796518 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:28.796518 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:28.796521 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:15:28.796578 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:28.796580 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.797523 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.797524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:28.798930 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:28.798932 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.799679 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.799680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:28.799763 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:28.799777 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 06:15:28.799863 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 06:15:28.799920 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":93,"threads_started":1}
I20260926 06:15:28.800117 10510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:28.795864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.193:37843 (local address 127.0.0.1:34980)
0926 06:15:28.796225 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:28.796236 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:28.796290 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:28.796604 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:28.796605 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:28.796608 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:15:28.796641 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:28.796643 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.797923 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.797924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:28.799021 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:28.799023 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.799858 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.799859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:28.799918 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:28.799921 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:15:28.799959 (+ 38us) client_negotiation.cc:240] Negotiation successful
0926 06:15:28.799992 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":30,"thread_start_us":70,"threads_started":1}
I20260926 06:15:28.800117 10512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:28.796108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34980 (local address 127.2.110.193:37843)
0926 06:15:28.796442 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:28.796446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:28.796455 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:15:28.796490 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:28.796492 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:28.796506 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:28.796713 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:15:28.796760 (+ 47us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.797837 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.799085 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.799789 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.799858 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:28.799934 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:28.799974 (+ 40us) server_negotiation.cc:299] Negotiation successful
0926 06:15:28.800014 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":100,"threads_started":1}
I20260926 06:15:28.800311 10509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:28.795798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33412 (local address 127.2.110.194:36115)
0926 06:15:28.796268 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:28.796273 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:28.796283 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:15:28.796345 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:28.796347 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:28.796362 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:28.796439 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:15:28.796627 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.797780 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.799002 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.799730 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.800060 (+ 330us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:28.800140 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:28.800166 (+ 26us) server_negotiation.cc:299] Negotiation successful
0926 06:15:28.800211 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":99,"threads_started":1}
I20260926 06:15:28.800748 10507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:28.795647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.194:36115 (local address 127.0.0.1:33412)
0926 06:15:28.795787 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:28.795835 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:28.795915 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:28.796460 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:28.796464 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:28.796467 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:15:28.796542 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:28.796544 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.797899 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.797900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:28.798915 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:28.798918 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.799856 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.799856 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:28.799933 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:28.799938 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 06:15:28.799970 (+ 32us) client_negotiation.cc:240] Negotiation successful
0926 06:15:28.800014 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":40,"thread_start_us":68,"threads_started":1}
I20260926 06:15:28.801121 10511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:28.796019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41970 (local address 127.2.110.195:34709)
0926 06:15:28.796178 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:28.796184 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:28.796198 (+ 14us) server_negotiation.cc:407] Connection header received
0926 06:15:28.796241 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:28.796243 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:28.796262 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:28.796351 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:15:28.796662 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.797417 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.799025 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.799570 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.799676 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:28.799766 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:28.800906 (+ 1140us) server_negotiation.cc:299] Negotiation successful
0926 06:15:28.801027 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":98,"threads_started":1}
I20260926 06:15:28.801141 10139 tablet_service.cc:1467] Processing CreateTablet for tablet c4596be22e7441feb732a5d60d178e6a (DEFAULT_TABLE table=test-table [id=3a019009dabd41408f5b441d7589c380]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:15:28.801607 10139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4596be22e7441feb732a5d60d178e6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:28.801860 10406 tablet_service.cc:1467] Processing CreateTablet for tablet c4596be22e7441feb732a5d60d178e6a (DEFAULT_TABLE table=test-table [id=3a019009dabd41408f5b441d7589c380]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:15:28.802171 10406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4596be22e7441feb732a5d60d178e6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:28.804850 10513 tablet_bootstrap.cc:492] T c4596be22e7441feb732a5d60d178e6a P 56036654e641486b99a44119fcff490f: Bootstrap starting.
I20260926 06:15:28.805401 10514 tablet_bootstrap.cc:492] T c4596be22e7441feb732a5d60d178e6a P df439d14be854ee5a671c9817319a4d4: Bootstrap starting.
I20260926 06:15:28.805567 10513 tablet_bootstrap.cc:654] T c4596be22e7441feb732a5d60d178e6a P 56036654e641486b99a44119fcff490f: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:28.805917 10513 log.cc:824] T c4596be22e7441feb732a5d60d178e6a P 56036654e641486b99a44119fcff490f: Log is configured to *not* fsync() on all Append() calls
I20260926 06:15:28.805863 10279 tablet_service.cc:1467] Processing CreateTablet for tablet c4596be22e7441feb732a5d60d178e6a (DEFAULT_TABLE table=test-table [id=3a019009dabd41408f5b441d7589c380]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:15:28.806025 10514 tablet_bootstrap.cc:654] T c4596be22e7441feb732a5d60d178e6a P df439d14be854ee5a671c9817319a4d4: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:28.806205 10279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4596be22e7441feb732a5d60d178e6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:28.806383 10514 log.cc:824] T c4596be22e7441feb732a5d60d178e6a P df439d14be854ee5a671c9817319a4d4: Log is configured to *not* fsync() on all Append() calls
I20260926 06:15:28.806777 10513 tablet_bootstrap.cc:492] T c4596be22e7441feb732a5d60d178e6a P 56036654e641486b99a44119fcff490f: No bootstrap required, opened a new log
I20260926 06:15:28.806869 10513 ts_tablet_manager.cc:1397] T c4596be22e7441feb732a5d60d178e6a P 56036654e641486b99a44119fcff490f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 06:15:28.807183 10514 tablet_bootstrap.cc:492] T c4596be22e7441feb732a5d60d178e6a P df439d14be854ee5a671c9817319a4d4: No bootstrap required, opened a new log
I20260926 06:15:28.807255 10514 ts_tablet_manager.cc:1397] T c4596be22e7441feb732a5d60d178e6a P df439d14be854ee5a671c9817319a4d4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 06:15:28.808995 10513 raft_consensus.cc:348] T c4596be22e7441feb732a5d60d178e6a P 56036654e641486b99a44119fcff490f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } } peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } }
I20260926 06:15:28.809183 10513 raft_consensus.cc:374] T c4596be22e7441feb732a5d60d178e6a P 56036654e641486b99a44119fcff490f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:28.809219 10513 raft_consensus.cc:729] T c4596be22e7441feb732a5d60d178e6a P 56036654e641486b99a44119fcff490f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56036654e641486b99a44119fcff490f, State: Initialized, Role: FOLLOWER
I20260926 06:15:28.809429 10513 consensus_queue.cc:260] T c4596be22e7441feb732a5d60d178e6a P 56036654e641486b99a44119fcff490f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } } peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } }
I20260926 06:15:28.809470 10514 raft_consensus.cc:348] T c4596be22e7441feb732a5d60d178e6a P df439d14be854ee5a671c9817319a4d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } } peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } }
I20260926 06:15:28.809621 10514 raft_consensus.cc:374] T c4596be22e7441feb732a5d60d178e6a P df439d14be854ee5a671c9817319a4d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:28.809660 10514 raft_consensus.cc:729] T c4596be22e7441feb732a5d60d178e6a P df439d14be854ee5a671c9817319a4d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df439d14be854ee5a671c9817319a4d4, State: Initialized, Role: FOLLOWER
I20260926 06:15:28.809749 10513 ts_tablet_manager.cc:1428] T c4596be22e7441feb732a5d60d178e6a P 56036654e641486b99a44119fcff490f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 06:15:28.809875 10205 heartbeater.cc:503] Master 127.2.110.254:46239 was elected leader, sending a full tablet report...
I20260926 06:15:28.809782 10514 consensus_queue.cc:260] T c4596be22e7441feb732a5d60d178e6a P df439d14be854ee5a671c9817319a4d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } } peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } }
I20260926 06:15:28.810465 10518 tablet_bootstrap.cc:492] T c4596be22e7441feb732a5d60d178e6a P b54d024ec6964d07844331e5a81db31e: Bootstrap starting.
I20260926 06:15:28.810678 10514 ts_tablet_manager.cc:1428] T c4596be22e7441feb732a5d60d178e6a P df439d14be854ee5a671c9817319a4d4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 06:15:28.810812 10485 heartbeater.cc:503] Master 127.2.110.254:46239 was elected leader, sending a full tablet report...
I20260926 06:15:28.811169 10518 tablet_bootstrap.cc:654] T c4596be22e7441feb732a5d60d178e6a P b54d024ec6964d07844331e5a81db31e: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:28.811537 10518 log.cc:824] T c4596be22e7441feb732a5d60d178e6a P b54d024ec6964d07844331e5a81db31e: Log is configured to *not* fsync() on all Append() calls
I20260926 06:15:28.812352 10518 tablet_bootstrap.cc:492] T c4596be22e7441feb732a5d60d178e6a P b54d024ec6964d07844331e5a81db31e: No bootstrap required, opened a new log
I20260926 06:15:28.812412 10518 ts_tablet_manager.cc:1397] T c4596be22e7441feb732a5d60d178e6a P b54d024ec6964d07844331e5a81db31e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 06:15:28.813946 10518 raft_consensus.cc:348] T c4596be22e7441feb732a5d60d178e6a P b54d024ec6964d07844331e5a81db31e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } } peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } }
I20260926 06:15:28.814086 10518 raft_consensus.cc:374] T c4596be22e7441feb732a5d60d178e6a P b54d024ec6964d07844331e5a81db31e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:28.814109 10518 raft_consensus.cc:729] T c4596be22e7441feb732a5d60d178e6a P b54d024ec6964d07844331e5a81db31e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b54d024ec6964d07844331e5a81db31e, State: Initialized, Role: FOLLOWER
I20260926 06:15:28.814244 10518 consensus_queue.cc:260] T c4596be22e7441feb732a5d60d178e6a P b54d024ec6964d07844331e5a81db31e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } } peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } }
I20260926 06:15:28.814491 10518 ts_tablet_manager.cc:1428] T c4596be22e7441feb732a5d60d178e6a P b54d024ec6964d07844331e5a81db31e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:15:28.814713 10345 heartbeater.cc:503] Master 127.2.110.254:46239 was elected leader, sending a full tablet report...
W20260926 06:15:28.820595 10346 tablet.cc:2369] T c4596be22e7441feb732a5d60d178e6a P b54d024ec6964d07844331e5a81db31e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:15:28.877372 10521 raft_consensus.cc:482] T c4596be22e7441feb732a5d60d178e6a P b54d024ec6964d07844331e5a81db31e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:15:28.877506 10521 raft_consensus.cc:504] T c4596be22e7441feb732a5d60d178e6a P b54d024ec6964d07844331e5a81db31e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } } peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } }
I20260926 06:15:28.878027 10521 leader_election.cc:288] T c4596be22e7441feb732a5d60d178e6a P b54d024ec6964d07844331e5a81db31e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56036654e641486b99a44119fcff490f (127.2.110.193:37843), df439d14be854ee5a671c9817319a4d4 (127.2.110.195:34709)
I20260926 06:15:28.880928 10512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:28.878292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:54395 (local address 127.2.110.193:37843)
0926 06:15:28.878339 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:28.878341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:28.878436 (+ 95us) server_negotiation.cc:407] Connection header received
0926 06:15:28.878467 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:28.878467 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:28.878476 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:28.878552 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:15:28.878706 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.879188 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.880155 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.880534 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.880621 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:28.880630 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:28.880817 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 06:15:28.880858 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 06:15:28.880942 10238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:28.878242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.193:37843 (local address 127.2.110.194:54395)
0926 06:15:28.878311 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:28.878324 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:28.878398 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:28.878559 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:28.878560 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:28.878561 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:15:28.878632 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:28.878634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.879291 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.879292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:28.880043 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:28.880045 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.880620 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.880621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:28.880751 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:28.880755 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:15:28.880812 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 06:15:28.880854 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 06:15:28.881018 10511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:28.878379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:34135 (local address 127.2.110.195:34709)
0926 06:15:28.878438 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:28.878441 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:28.878549 (+ 108us) server_negotiation.cc:407] Connection header received
0926 06:15:28.878618 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:28.878619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:28.878629 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:28.878712 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:15:28.878885 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.879345 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.880281 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.880647 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.880748 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:28.880757 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:28.880909 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 06:15:28.880945 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 06:15:28.881160 10522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:28.878282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.195:34709 (local address 127.2.110.194:34135)
0926 06:15:28.878471 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:28.878487 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:28.878563 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:28.878723 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:28.878724 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:28.878725 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:15:28.878805 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:28.878806 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.879429 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.879430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:28.880188 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:28.880190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.880742 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.880743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:28.880824 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:28.880827 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:15:28.880891 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 06:15:28.880932 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":96,"threads_started":1}
I20260926 06:15:28.881309 10159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4596be22e7441feb732a5d60d178e6a" candidate_uuid: "b54d024ec6964d07844331e5a81db31e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56036654e641486b99a44119fcff490f" is_pre_election: true
I20260926 06:15:28.881517 10159 raft_consensus.cc:2454] T c4596be22e7441feb732a5d60d178e6a P 56036654e641486b99a44119fcff490f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b54d024ec6964d07844331e5a81db31e in term 0.
I20260926 06:15:28.881711 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4596be22e7441feb732a5d60d178e6a" candidate_uuid: "b54d024ec6964d07844331e5a81db31e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df439d14be854ee5a671c9817319a4d4" is_pre_election: true
I20260926 06:15:28.881772 10230 leader_election.cc:302] T c4596be22e7441feb732a5d60d178e6a P b54d024ec6964d07844331e5a81db31e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56036654e641486b99a44119fcff490f, b54d024ec6964d07844331e5a81db31e; no voters:
I20260926 06:15:28.881903 10438 raft_consensus.cc:2454] T c4596be22e7441feb732a5d60d178e6a P df439d14be854ee5a671c9817319a4d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b54d024ec6964d07844331e5a81db31e in term 0.
I20260926 06:15:28.881923 10521 raft_consensus.cc:2781] T c4596be22e7441feb732a5d60d178e6a P b54d024ec6964d07844331e5a81db31e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:15:28.881971 10521 raft_consensus.cc:482] T c4596be22e7441feb732a5d60d178e6a P b54d024ec6964d07844331e5a81db31e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:15:28.882005 10521 raft_consensus.cc:3037] T c4596be22e7441feb732a5d60d178e6a P b54d024ec6964d07844331e5a81db31e [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:28.882961 10521 raft_consensus.cc:504] T c4596be22e7441feb732a5d60d178e6a P b54d024ec6964d07844331e5a81db31e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } } peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } }
I20260926 06:15:28.883131 10521 leader_election.cc:288] T c4596be22e7441feb732a5d60d178e6a P b54d024ec6964d07844331e5a81db31e [CANDIDATE]: Term 1 election: Requested vote from peers 56036654e641486b99a44119fcff490f (127.2.110.193:37843), df439d14be854ee5a671c9817319a4d4 (127.2.110.195:34709)
I20260926 06:15:28.883307 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4596be22e7441feb732a5d60d178e6a" candidate_uuid: "b54d024ec6964d07844331e5a81db31e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df439d14be854ee5a671c9817319a4d4"
I20260926 06:15:28.883399 10438 raft_consensus.cc:3037] T c4596be22e7441feb732a5d60d178e6a P df439d14be854ee5a671c9817319a4d4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:28.883441 10159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4596be22e7441feb732a5d60d178e6a" candidate_uuid: "b54d024ec6964d07844331e5a81db31e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56036654e641486b99a44119fcff490f"
I20260926 06:15:28.883508 10159 raft_consensus.cc:3037] T c4596be22e7441feb732a5d60d178e6a P 56036654e641486b99a44119fcff490f [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:28.884059 10438 raft_consensus.cc:2454] T c4596be22e7441feb732a5d60d178e6a P df439d14be854ee5a671c9817319a4d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b54d024ec6964d07844331e5a81db31e in term 1.
I20260926 06:15:28.884236 10159 raft_consensus.cc:2454] T c4596be22e7441feb732a5d60d178e6a P 56036654e641486b99a44119fcff490f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b54d024ec6964d07844331e5a81db31e in term 1.
I20260926 06:15:28.884232 10231 leader_election.cc:302] T c4596be22e7441feb732a5d60d178e6a P b54d024ec6964d07844331e5a81db31e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b54d024ec6964d07844331e5a81db31e, df439d14be854ee5a671c9817319a4d4; no voters:
I20260926 06:15:28.884377 10521 raft_consensus.cc:2781] T c4596be22e7441feb732a5d60d178e6a P b54d024ec6964d07844331e5a81db31e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:15:28.884565 10521 raft_consensus.cc:686] T c4596be22e7441feb732a5d60d178e6a P b54d024ec6964d07844331e5a81db31e [term 1 LEADER]: Becoming Leader. State: Replica: b54d024ec6964d07844331e5a81db31e, State: Running, Role: LEADER
I20260926 06:15:28.884686 10521 consensus_queue.cc:237] T c4596be22e7441feb732a5d60d178e6a P b54d024ec6964d07844331e5a81db31e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } } peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } }
I20260926 06:15:28.885608 10002 catalog_manager.cc:5495] T c4596be22e7441feb732a5d60d178e6a P b54d024ec6964d07844331e5a81db31e reported cstate change: term changed from 0 to 1, leader changed from <none> to b54d024ec6964d07844331e5a81db31e (127.2.110.194). New cstate: current_term: 1 leader_uuid: "b54d024ec6964d07844331e5a81db31e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } health_report { overall_health: UNKNOWN } } }
I20260926 06:15:28.899214 10509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:28.896707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33428 (local address 127.2.110.194:36115)
0926 06:15:28.896789 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:28.896791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:28.896798 (+ 7us) server_negotiation.cc:407] Connection header received
0926 06:15:28.896833 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:28.896834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:28.896851 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:28.896937 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:15:28.897097 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.897640 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.898414 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.898606 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.898694 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:28.898876 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:15:28.898985 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:15:28.899082 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:28.899101 (+ 19us) server_negotiation.cc:299] Negotiation successful
0926 06:15:28.899124 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 06:15:28.901326 10159 raft_consensus.cc:1261] T c4596be22e7441feb732a5d60d178e6a P 56036654e641486b99a44119fcff490f [term 1 FOLLOWER]: Refusing update from remote peer b54d024ec6964d07844331e5a81db31e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:15:28.901448 10438 raft_consensus.cc:1261] T c4596be22e7441feb732a5d60d178e6a P df439d14be854ee5a671c9817319a4d4 [term 1 FOLLOWER]: Refusing update from remote peer b54d024ec6964d07844331e5a81db31e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:15:28.901850 10523 consensus_queue.cc:1035] T c4596be22e7441feb732a5d60d178e6a P b54d024ec6964d07844331e5a81db31e [LEADER]: Connected to new peer: Peer: permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:15:28.902000 10521 consensus_queue.cc:1035] T c4596be22e7441feb732a5d60d178e6a P b54d024ec6964d07844331e5a81db31e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:15:28.904578 10526 mvcc.cc:205] Tried to move back new op lower bound from 7333492035176034304 to 7333492035112370176. Current Snapshot: MvccSnapshot[applied={T|T < 7333492035176034304}]
I20260926 06:15:28.904654 10527 mvcc.cc:205] Tried to move back new op lower bound from 7333492035176034304 to 7333492035112370176. Current Snapshot: MvccSnapshot[applied={T|T < 7333492035176034304}]
I20260926 06:15:28.905260 10528 mvcc.cc:205] Tried to move back new op lower bound from 7333492035176034304 to 7333492035112370176. Current Snapshot: MvccSnapshot[applied={T|T < 7333492035176034304}]
I20260926 06:15:28.907717 10002 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:15:28.907864 10002 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:15:28.909006 10002 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 24536bc278a44d40bfcee7eb68808bf6: Sending DeleteTablet for 3 replicas of tablet c4596be22e7441feb732a5d60d178e6a
I20260926 06:15:28.909591 10279 tablet_service.cc:1514] Processing DeleteTablet for tablet c4596be22e7441feb732a5d60d178e6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:15:28 UTC) from {username='oryx', principal='oryx/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:33412
I20260926 06:15:28.909647 10139 tablet_service.cc:1514] Processing DeleteTablet for tablet c4596be22e7441feb732a5d60d178e6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:15:28 UTC) from {username='oryx', principal='oryx/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:34980
I20260926 06:15:28.909857 10535 tablet_replica.cc:330] T c4596be22e7441feb732a5d60d178e6a P b54d024ec6964d07844331e5a81db31e: stopping tablet replica
I20260926 06:15:28.909972 10535 raft_consensus.cc:2229] T c4596be22e7441feb732a5d60d178e6a P b54d024ec6964d07844331e5a81db31e [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:15:28.910030 10406 tablet_service.cc:1514] Processing DeleteTablet for tablet c4596be22e7441feb732a5d60d178e6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:15:28 UTC) from {username='oryx', principal='oryx/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:41970
I20260926 06:15:28.910115 10535 raft_consensus.cc:2258] T c4596be22e7441feb732a5d60d178e6a P b54d024ec6964d07844331e5a81db31e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:15:28.910043 10002 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53648:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:15:28.910332 10002 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:15:28.910518 10537 tablet_replica.cc:330] T c4596be22e7441feb732a5d60d178e6a P df439d14be854ee5a671c9817319a4d4: stopping tablet replica
I20260926 06:15:28.910610 10535 ts_tablet_manager.cc:1905] T c4596be22e7441feb732a5d60d178e6a P b54d024ec6964d07844331e5a81db31e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:15:28.910724 10537 raft_consensus.cc:2229] T c4596be22e7441feb732a5d60d178e6a P df439d14be854ee5a671c9817319a4d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:15:28.911140 10537 raft_consensus.cc:2258] T c4596be22e7441feb732a5d60d178e6a P df439d14be854ee5a671c9817319a4d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:15:28.911671 10537 ts_tablet_manager.cc:1905] T c4596be22e7441feb732a5d60d178e6a P df439d14be854ee5a671c9817319a4d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:15:28.912091 10279 tablet_service.cc:1467] Processing CreateTablet for tablet 413b877177164ab293886930864c9329 (DEFAULT_TABLE table=test-table [id=27b0ca7b1dea434bad28f27f3d0ffd31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:15:28.912232 10279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 413b877177164ab293886930864c9329. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:28.912644 10406 tablet_service.cc:1467] Processing CreateTablet for tablet 413b877177164ab293886930864c9329 (DEFAULT_TABLE table=test-table [id=27b0ca7b1dea434bad28f27f3d0ffd31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:15:28.912704 10139 tablet_service.cc:1467] Processing CreateTablet for tablet 413b877177164ab293886930864c9329 (DEFAULT_TABLE table=test-table [id=27b0ca7b1dea434bad28f27f3d0ffd31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:15:28.912744 10406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 413b877177164ab293886930864c9329. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:28.912828 10139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 413b877177164ab293886930864c9329. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:28.913671 10518 tablet_bootstrap.cc:492] T 413b877177164ab293886930864c9329 P b54d024ec6964d07844331e5a81db31e: Bootstrap starting.
I20260926 06:15:28.914103 10518 tablet_bootstrap.cc:654] T 413b877177164ab293886930864c9329 P b54d024ec6964d07844331e5a81db31e: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:28.914121 10514 tablet_bootstrap.cc:492] T 413b877177164ab293886930864c9329 P df439d14be854ee5a671c9817319a4d4: Bootstrap starting.
I20260926 06:15:28.914255 10513 tablet_bootstrap.cc:492] T 413b877177164ab293886930864c9329 P 56036654e641486b99a44119fcff490f: Bootstrap starting.
I20260926 06:15:28.914548 10514 tablet_bootstrap.cc:654] T 413b877177164ab293886930864c9329 P df439d14be854ee5a671c9817319a4d4: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:28.914651 10513 tablet_bootstrap.cc:654] T 413b877177164ab293886930864c9329 P 56036654e641486b99a44119fcff490f: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:28.914652 10535 ts_tablet_manager.cc:1918] T c4596be22e7441feb732a5d60d178e6a P b54d024ec6964d07844331e5a81db31e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:15:28.914726 10535 log.cc:1196] T c4596be22e7441feb732a5d60d178e6a P b54d024ec6964d07844331e5a81db31e: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-1/wal/wals/c4596be22e7441feb732a5d60d178e6a
I20260926 06:15:28.915040 10535 ts_tablet_manager.cc:1939] T c4596be22e7441feb732a5d60d178e6a P b54d024ec6964d07844331e5a81db31e: Deleting consensus metadata
I20260926 06:15:28.915472 9986 catalog_manager.cc:4841] TS b54d024ec6964d07844331e5a81db31e (127.2.110.194:36115): tablet c4596be22e7441feb732a5d60d178e6a (table test-table [id=3a019009dabd41408f5b441d7589c380]) successfully deleted
I20260926 06:15:28.915633 10536 tablet_replica.cc:330] T c4596be22e7441feb732a5d60d178e6a P 56036654e641486b99a44119fcff490f: stopping tablet replica
I20260926 06:15:28.915761 10536 raft_consensus.cc:2229] T c4596be22e7441feb732a5d60d178e6a P 56036654e641486b99a44119fcff490f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:15:28.915882 10536 raft_consensus.cc:2258] T c4596be22e7441feb732a5d60d178e6a P 56036654e641486b99a44119fcff490f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:15:28.915975 10513 tablet_bootstrap.cc:492] T 413b877177164ab293886930864c9329 P 56036654e641486b99a44119fcff490f: No bootstrap required, opened a new log
I20260926 06:15:28.916026 10513 ts_tablet_manager.cc:1397] T 413b877177164ab293886930864c9329 P 56036654e641486b99a44119fcff490f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 06:15:28.916146 10537 ts_tablet_manager.cc:1918] T c4596be22e7441feb732a5d60d178e6a P df439d14be854ee5a671c9817319a4d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:15:28.916188 10513 raft_consensus.cc:348] T 413b877177164ab293886930864c9329 P 56036654e641486b99a44119fcff490f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } } peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } }
I20260926 06:15:28.916281 10513 raft_consensus.cc:374] T 413b877177164ab293886930864c9329 P 56036654e641486b99a44119fcff490f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:28.916311 10513 raft_consensus.cc:729] T 413b877177164ab293886930864c9329 P 56036654e641486b99a44119fcff490f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56036654e641486b99a44119fcff490f, State: Initialized, Role: FOLLOWER
I20260926 06:15:28.916271 10537 log.cc:1196] T c4596be22e7441feb732a5d60d178e6a P df439d14be854ee5a671c9817319a4d4: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-2/wal/wals/c4596be22e7441feb732a5d60d178e6a
I20260926 06:15:28.916406 10513 consensus_queue.cc:260] T 413b877177164ab293886930864c9329 P 56036654e641486b99a44119fcff490f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } } peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } }
I20260926 06:15:28.916522 10513 ts_tablet_manager.cc:1428] T 413b877177164ab293886930864c9329 P 56036654e641486b99a44119fcff490f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 06:15:28.916667 10518 tablet_bootstrap.cc:492] T 413b877177164ab293886930864c9329 P b54d024ec6964d07844331e5a81db31e: No bootstrap required, opened a new log
I20260926 06:15:28.916726 10518 ts_tablet_manager.cc:1397] T 413b877177164ab293886930864c9329 P b54d024ec6964d07844331e5a81db31e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 06:15:28.916297 10536 ts_tablet_manager.cc:1905] T c4596be22e7441feb732a5d60d178e6a P 56036654e641486b99a44119fcff490f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:15:28.916900 10537 ts_tablet_manager.cc:1939] T c4596be22e7441feb732a5d60d178e6a P df439d14be854ee5a671c9817319a4d4: Deleting consensus metadata
I20260926 06:15:28.916904 10518 raft_consensus.cc:348] T 413b877177164ab293886930864c9329 P b54d024ec6964d07844331e5a81db31e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } } peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } }
I20260926 06:15:28.916985 10518 raft_consensus.cc:374] T 413b877177164ab293886930864c9329 P b54d024ec6964d07844331e5a81db31e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:28.917012 10518 raft_consensus.cc:729] T 413b877177164ab293886930864c9329 P b54d024ec6964d07844331e5a81db31e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b54d024ec6964d07844331e5a81db31e, State: Initialized, Role: FOLLOWER
I20260926 06:15:28.917074 10518 consensus_queue.cc:260] T 413b877177164ab293886930864c9329 P b54d024ec6964d07844331e5a81db31e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } } peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } }
I20260926 06:15:28.917167 10518 ts_tablet_manager.cc:1428] T 413b877177164ab293886930864c9329 P b54d024ec6964d07844331e5a81db31e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:28.917668 9987 catalog_manager.cc:4841] TS df439d14be854ee5a671c9817319a4d4 (127.2.110.195:34709): tablet c4596be22e7441feb732a5d60d178e6a (table test-table [id=3a019009dabd41408f5b441d7589c380]) successfully deleted
I20260926 06:15:28.918125 10514 tablet_bootstrap.cc:492] T 413b877177164ab293886930864c9329 P df439d14be854ee5a671c9817319a4d4: No bootstrap required, opened a new log
I20260926 06:15:28.918205 10514 ts_tablet_manager.cc:1397] T 413b877177164ab293886930864c9329 P df439d14be854ee5a671c9817319a4d4: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260926 06:15:28.918358 10514 raft_consensus.cc:348] T 413b877177164ab293886930864c9329 P df439d14be854ee5a671c9817319a4d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } } peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } }
I20260926 06:15:28.918455 10514 raft_consensus.cc:374] T 413b877177164ab293886930864c9329 P df439d14be854ee5a671c9817319a4d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:28.918484 10514 raft_consensus.cc:729] T 413b877177164ab293886930864c9329 P df439d14be854ee5a671c9817319a4d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df439d14be854ee5a671c9817319a4d4, State: Initialized, Role: FOLLOWER
I20260926 06:15:28.918532 10514 consensus_queue.cc:260] T 413b877177164ab293886930864c9329 P df439d14be854ee5a671c9817319a4d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } } peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } }
I20260926 06:15:28.918625 10514 ts_tablet_manager.cc:1428] T 413b877177164ab293886930864c9329 P df439d14be854ee5a671c9817319a4d4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:28.918691 10536 ts_tablet_manager.cc:1918] T c4596be22e7441feb732a5d60d178e6a P 56036654e641486b99a44119fcff490f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:15:28.918763 10536 log.cc:1196] T c4596be22e7441feb732a5d60d178e6a P 56036654e641486b99a44119fcff490f: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-0/wal/wals/c4596be22e7441feb732a5d60d178e6a
I20260926 06:15:28.919081 10536 ts_tablet_manager.cc:1939] T c4596be22e7441feb732a5d60d178e6a P 56036654e641486b99a44119fcff490f: Deleting consensus metadata
I20260926 06:15:28.919492 9986 catalog_manager.cc:4841] TS 56036654e641486b99a44119fcff490f (127.2.110.193:37843): tablet c4596be22e7441feb732a5d60d178e6a (table test-table [id=3a019009dabd41408f5b441d7589c380]) successfully deleted
I20260926 06:15:28.961158 10519 raft_consensus.cc:482] T 413b877177164ab293886930864c9329 P df439d14be854ee5a671c9817319a4d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:15:28.961319 10519 raft_consensus.cc:504] T 413b877177164ab293886930864c9329 P df439d14be854ee5a671c9817319a4d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } } peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } }
I20260926 06:15:28.961934 10519 leader_election.cc:288] T 413b877177164ab293886930864c9329 P df439d14be854ee5a671c9817319a4d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56036654e641486b99a44119fcff490f (127.2.110.193:37843), b54d024ec6964d07844331e5a81db31e (127.2.110.194:36115)
I20260926 06:15:28.966434 10509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:28.962489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:34491 (local address 127.2.110.194:36115)
0926 06:15:28.962542 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:28.962544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:28.962761 (+ 217us) server_negotiation.cc:407] Connection header received
0926 06:15:28.962879 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:28.962880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:28.962890 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:28.962920 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:15:28.963217 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.963938 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.965172 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.965916 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.966050 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:28.966064 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:28.966283 (+ 219us) server_negotiation.cc:299] Negotiation successful
0926 06:15:28.966337 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 06:15:28.966609 10512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:28.962299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:36555 (local address 127.2.110.193:37843)
0926 06:15:28.962353 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:28.962356 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:28.962565 (+ 209us) server_negotiation.cc:407] Connection header received
0926 06:15:28.962679 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:28.962681 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:28.962697 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:28.962758 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:15:28.963025 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.963841 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.965418 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.966129 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.966246 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:28.966256 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:28.966488 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 06:15:28.966534 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 06:15:28.966614 10378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:28.962248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.193:37843 (local address 127.2.110.195:36555)
0926 06:15:28.962419 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:28.962434 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:28.962545 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:28.962821 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:28.962822 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:28.962824 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:15:28.962922 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:28.962924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.963961 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.963962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:28.965265 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:28.965268 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.966259 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.966261 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:28.966365 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:28.966369 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:15:28.966475 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 06:15:28.966530 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144}
I20260926 06:15:28.966929 10159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "413b877177164ab293886930864c9329" candidate_uuid: "df439d14be854ee5a671c9817319a4d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56036654e641486b99a44119fcff490f" is_pre_election: true
I20260926 06:15:28.967034 10541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:28.962374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.194:36115 (local address 127.2.110.195:34491)
0926 06:15:28.962667 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:28.962681 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:28.962796 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:28.962996 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:28.962997 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:28.962999 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:15:28.963112 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:28.963115 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.964155 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.964156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:28.965061 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:28.965064 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.966078 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.966080 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:28.966159 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:28.966165 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 06:15:28.966896 (+ 731us) client_negotiation.cc:240] Negotiation successful
0926 06:15:28.966952 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":94,"threads_started":1}
I20260926 06:15:28.967046 10159 raft_consensus.cc:2454] T 413b877177164ab293886930864c9329 P 56036654e641486b99a44119fcff490f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df439d14be854ee5a671c9817319a4d4 in term 0.
I20260926 06:15:28.967278 10370 leader_election.cc:302] T 413b877177164ab293886930864c9329 P df439d14be854ee5a671c9817319a4d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56036654e641486b99a44119fcff490f, df439d14be854ee5a671c9817319a4d4; no voters:
I20260926 06:15:28.967262 10298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "413b877177164ab293886930864c9329" candidate_uuid: "df439d14be854ee5a671c9817319a4d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b54d024ec6964d07844331e5a81db31e" is_pre_election: true
I20260926 06:15:28.967409 10298 raft_consensus.cc:2454] T 413b877177164ab293886930864c9329 P b54d024ec6964d07844331e5a81db31e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df439d14be854ee5a671c9817319a4d4 in term 0.
I20260926 06:15:28.967465 10519 raft_consensus.cc:2781] T 413b877177164ab293886930864c9329 P df439d14be854ee5a671c9817319a4d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:15:28.967514 10519 raft_consensus.cc:482] T 413b877177164ab293886930864c9329 P df439d14be854ee5a671c9817319a4d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:15:28.967551 10519 raft_consensus.cc:3037] T 413b877177164ab293886930864c9329 P df439d14be854ee5a671c9817319a4d4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:28.968348 10519 raft_consensus.cc:504] T 413b877177164ab293886930864c9329 P df439d14be854ee5a671c9817319a4d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } } peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } }
I20260926 06:15:28.968521 10519 leader_election.cc:288] T 413b877177164ab293886930864c9329 P df439d14be854ee5a671c9817319a4d4 [CANDIDATE]: Term 1 election: Requested vote from peers 56036654e641486b99a44119fcff490f (127.2.110.193:37843), b54d024ec6964d07844331e5a81db31e (127.2.110.194:36115)
I20260926 06:15:28.968707 10298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "413b877177164ab293886930864c9329" candidate_uuid: "df439d14be854ee5a671c9817319a4d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b54d024ec6964d07844331e5a81db31e"
I20260926 06:15:28.968707 10159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "413b877177164ab293886930864c9329" candidate_uuid: "df439d14be854ee5a671c9817319a4d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56036654e641486b99a44119fcff490f"
I20260926 06:15:28.968784 10298 raft_consensus.cc:3037] T 413b877177164ab293886930864c9329 P b54d024ec6964d07844331e5a81db31e [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:28.968813 10159 raft_consensus.cc:3037] T 413b877177164ab293886930864c9329 P 56036654e641486b99a44119fcff490f [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:28.969619 10298 raft_consensus.cc:2454] T 413b877177164ab293886930864c9329 P b54d024ec6964d07844331e5a81db31e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df439d14be854ee5a671c9817319a4d4 in term 1.
I20260926 06:15:28.969619 10159 raft_consensus.cc:2454] T 413b877177164ab293886930864c9329 P 56036654e641486b99a44119fcff490f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df439d14be854ee5a671c9817319a4d4 in term 1.
I20260926 06:15:28.969831 10370 leader_election.cc:302] T 413b877177164ab293886930864c9329 P df439d14be854ee5a671c9817319a4d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56036654e641486b99a44119fcff490f, df439d14be854ee5a671c9817319a4d4; no voters:
I20260926 06:15:28.969987 10519 raft_consensus.cc:2781] T 413b877177164ab293886930864c9329 P df439d14be854ee5a671c9817319a4d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:15:28.970175 10519 raft_consensus.cc:686] T 413b877177164ab293886930864c9329 P df439d14be854ee5a671c9817319a4d4 [term 1 LEADER]: Becoming Leader. State: Replica: df439d14be854ee5a671c9817319a4d4, State: Running, Role: LEADER
I20260926 06:15:28.970257 10519 consensus_queue.cc:237] T 413b877177164ab293886930864c9329 P df439d14be854ee5a671c9817319a4d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } } peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } }
I20260926 06:15:28.971347 10002 catalog_manager.cc:5495] T 413b877177164ab293886930864c9329 P df439d14be854ee5a671c9817319a4d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to df439d14be854ee5a671c9817319a4d4 (127.2.110.195). New cstate: current_term: 1 leader_uuid: "df439d14be854ee5a671c9817319a4d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } health_report { overall_health: HEALTHY } } }
Sep 26 06:15:28 dist-test-slave-hndd krb5kdc[9957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403328, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.110.254@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM
W20260926 06:15:29.001972 10486 tablet.cc:2369] T 413b877177164ab293886930864c9329 P df439d14be854ee5a671c9817319a4d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:15:29.004531 10052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:28.994783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53654 (local address 127.2.110.254:46239)
0926 06:15:28.994867 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:28.994870 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:28.995122 (+ 252us) server_negotiation.cc:407] Connection header received
0926 06:15:28.995208 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:28.995209 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:28.995220 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:28.995287 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:28.996194 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.996787 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.997922 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.998105 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.998172 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:29.000833 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:29.000835 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:29.000835 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:29.000842 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:29.002900 (+ 2058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:29.003460 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:29.003461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:29.003461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:29.003528 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:29.003763 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:29.003763 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:29.003764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:29.003975 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:29.004050 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:29.004264 (+ 214us) server_negotiation.cc:299] Negotiation successful
0926 06:15:29.004420 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":76,"server-negotiator.queue_time_us":49}
I20260926 06:15:29.004549 10549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:28.994709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:46239 (local address 127.0.0.1:53654)
0926 06:15:28.994931 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:28.994945 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:28.995128 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:28.995315 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:28.995316 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:28.995833 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:28.995915 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:28.995919 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.996908 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.996909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:28.997810 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:28.997814 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:28.998196 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:28.998197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:28.998262 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:28.998938 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:28.998944 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:29.000702 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:29.003029 (+ 2327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:29.003037 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:29.003040 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:29.003341 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:29.003614 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:29.003615 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:29.003615 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:29.003691 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:29.004070 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:29.004077 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:29.004136 (+ 59us) client_negotiation.cc:769] Sending connection context
0926 06:15:29.004223 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 06:15:29.004366 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":129,"threads_started":1}
I20260926 06:15:29.007200 10002 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:53654:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 06:15:29.007465 10002 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:15:29.009032 10139 tablet_service.cc:1467] Processing CreateTablet for tablet ca4a988745be4211b6f48d3d339dc5c6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ed0ceee0932f41faa62680a891828337]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 06:15:29.009052 10279 tablet_service.cc:1467] Processing CreateTablet for tablet ca4a988745be4211b6f48d3d339dc5c6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ed0ceee0932f41faa62680a891828337]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 06:15:29.009208 10139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca4a988745be4211b6f48d3d339dc5c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:29.009217 10279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca4a988745be4211b6f48d3d339dc5c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:29.010517 10518 tablet_bootstrap.cc:492] T ca4a988745be4211b6f48d3d339dc5c6 P b54d024ec6964d07844331e5a81db31e: Bootstrap starting.
I20260926 06:15:29.010793 10406 tablet_service.cc:1467] Processing CreateTablet for tablet ca4a988745be4211b6f48d3d339dc5c6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ed0ceee0932f41faa62680a891828337]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 06:15:29.011076 10406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca4a988745be4211b6f48d3d339dc5c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:29.011227 10513 tablet_bootstrap.cc:492] T ca4a988745be4211b6f48d3d339dc5c6 P 56036654e641486b99a44119fcff490f: Bootstrap starting.
I20260926 06:15:29.011540 10518 tablet_bootstrap.cc:654] T ca4a988745be4211b6f48d3d339dc5c6 P b54d024ec6964d07844331e5a81db31e: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:29.011705 10513 tablet_bootstrap.cc:654] T ca4a988745be4211b6f48d3d339dc5c6 P 56036654e641486b99a44119fcff490f: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:29.012594 10513 tablet_bootstrap.cc:492] T ca4a988745be4211b6f48d3d339dc5c6 P 56036654e641486b99a44119fcff490f: No bootstrap required, opened a new log
I20260926 06:15:29.012657 10513 ts_tablet_manager.cc:1397] T ca4a988745be4211b6f48d3d339dc5c6 P 56036654e641486b99a44119fcff490f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:29.012805 10513 raft_consensus.cc:348] T ca4a988745be4211b6f48d3d339dc5c6 P 56036654e641486b99a44119fcff490f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } } peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } }
I20260926 06:15:29.012892 10513 raft_consensus.cc:374] T ca4a988745be4211b6f48d3d339dc5c6 P 56036654e641486b99a44119fcff490f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:29.012913 10513 raft_consensus.cc:729] T ca4a988745be4211b6f48d3d339dc5c6 P 56036654e641486b99a44119fcff490f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56036654e641486b99a44119fcff490f, State: Initialized, Role: FOLLOWER
I20260926 06:15:29.012966 10513 consensus_queue.cc:260] T ca4a988745be4211b6f48d3d339dc5c6 P 56036654e641486b99a44119fcff490f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } } peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } }
I20260926 06:15:29.013065 10513 ts_tablet_manager.cc:1428] T ca4a988745be4211b6f48d3d339dc5c6 P 56036654e641486b99a44119fcff490f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 06:15:29.013233 10517 tablet_replica.cc:439] T ca4a988745be4211b6f48d3d339dc5c6 P 56036654e641486b99a44119fcff490f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } } peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } } }
I20260926 06:15:29.013329 10517 tablet_replica.cc:442] T ca4a988745be4211b6f48d3d339dc5c6 P 56036654e641486b99a44119fcff490f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:15:29.014415 10514 tablet_bootstrap.cc:492] T ca4a988745be4211b6f48d3d339dc5c6 P df439d14be854ee5a671c9817319a4d4: Bootstrap starting.
I20260926 06:15:29.015165 10514 tablet_bootstrap.cc:654] T ca4a988745be4211b6f48d3d339dc5c6 P df439d14be854ee5a671c9817319a4d4: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:29.015442 10518 tablet_bootstrap.cc:492] T ca4a988745be4211b6f48d3d339dc5c6 P b54d024ec6964d07844331e5a81db31e: No bootstrap required, opened a new log
I20260926 06:15:29.015498 10518 ts_tablet_manager.cc:1397] T ca4a988745be4211b6f48d3d339dc5c6 P b54d024ec6964d07844331e5a81db31e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260926 06:15:29.015667 10518 raft_consensus.cc:348] T ca4a988745be4211b6f48d3d339dc5c6 P b54d024ec6964d07844331e5a81db31e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } } peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } }
I20260926 06:15:29.015754 10518 raft_consensus.cc:374] T ca4a988745be4211b6f48d3d339dc5c6 P b54d024ec6964d07844331e5a81db31e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:29.015776 10518 raft_consensus.cc:729] T ca4a988745be4211b6f48d3d339dc5c6 P b54d024ec6964d07844331e5a81db31e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b54d024ec6964d07844331e5a81db31e, State: Initialized, Role: FOLLOWER
I20260926 06:15:29.015827 10518 consensus_queue.cc:260] T ca4a988745be4211b6f48d3d339dc5c6 P b54d024ec6964d07844331e5a81db31e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } } peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } }
I20260926 06:15:29.015933 10518 ts_tablet_manager.cc:1428] T ca4a988745be4211b6f48d3d339dc5c6 P b54d024ec6964d07844331e5a81db31e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 06:15:29.016064 10523 tablet_replica.cc:439] T ca4a988745be4211b6f48d3d339dc5c6 P b54d024ec6964d07844331e5a81db31e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } } peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } } }
I20260926 06:15:29.016161 10523 tablet_replica.cc:442] T ca4a988745be4211b6f48d3d339dc5c6 P b54d024ec6964d07844331e5a81db31e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:15:29.016510 10514 tablet_bootstrap.cc:492] T ca4a988745be4211b6f48d3d339dc5c6 P df439d14be854ee5a671c9817319a4d4: No bootstrap required, opened a new log
I20260926 06:15:29.016567 10514 ts_tablet_manager.cc:1397] T ca4a988745be4211b6f48d3d339dc5c6 P df439d14be854ee5a671c9817319a4d4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 06:15:29.016768 10514 raft_consensus.cc:348] T ca4a988745be4211b6f48d3d339dc5c6 P df439d14be854ee5a671c9817319a4d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } } peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } }
I20260926 06:15:29.016844 10514 raft_consensus.cc:374] T ca4a988745be4211b6f48d3d339dc5c6 P df439d14be854ee5a671c9817319a4d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:29.016857 10514 raft_consensus.cc:729] T ca4a988745be4211b6f48d3d339dc5c6 P df439d14be854ee5a671c9817319a4d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df439d14be854ee5a671c9817319a4d4, State: Initialized, Role: FOLLOWER
I20260926 06:15:29.016894 10514 consensus_queue.cc:260] T ca4a988745be4211b6f48d3d339dc5c6 P df439d14be854ee5a671c9817319a4d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } } peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } }
I20260926 06:15:29.016990 10514 ts_tablet_manager.cc:1428] T ca4a988745be4211b6f48d3d339dc5c6 P df439d14be854ee5a671c9817319a4d4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:29.017045 10519 tablet_replica.cc:439] T ca4a988745be4211b6f48d3d339dc5c6 P df439d14be854ee5a671c9817319a4d4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } } peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } } }
I20260926 06:15:29.017123 10519 tablet_replica.cc:442] T ca4a988745be4211b6f48d3d339dc5c6 P df439d14be854ee5a671c9817319a4d4: This TxnStatusTablet replica's current role is: FOLLOWER
W20260926 06:15:29.157217 10206 tablet.cc:2369] T 413b877177164ab293886930864c9329 P 56036654e641486b99a44119fcff490f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:15:29.272941 10519 raft_consensus.cc:482] T ca4a988745be4211b6f48d3d339dc5c6 P df439d14be854ee5a671c9817319a4d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:15:29.273047 10519 raft_consensus.cc:504] T ca4a988745be4211b6f48d3d339dc5c6 P df439d14be854ee5a671c9817319a4d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } } peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } }
I20260926 06:15:29.273180 10519 leader_election.cc:288] T ca4a988745be4211b6f48d3d339dc5c6 P df439d14be854ee5a671c9817319a4d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56036654e641486b99a44119fcff490f (127.2.110.193:37843), b54d024ec6964d07844331e5a81db31e (127.2.110.194:36115)
I20260926 06:15:29.273590 10298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca4a988745be4211b6f48d3d339dc5c6" candidate_uuid: "df439d14be854ee5a671c9817319a4d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b54d024ec6964d07844331e5a81db31e" is_pre_election: true
I20260926 06:15:29.273682 10159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca4a988745be4211b6f48d3d339dc5c6" candidate_uuid: "df439d14be854ee5a671c9817319a4d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56036654e641486b99a44119fcff490f" is_pre_election: true
I20260926 06:15:29.273741 10298 raft_consensus.cc:2454] T ca4a988745be4211b6f48d3d339dc5c6 P b54d024ec6964d07844331e5a81db31e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df439d14be854ee5a671c9817319a4d4 in term 0.
I20260926 06:15:29.273778 10159 raft_consensus.cc:2454] T ca4a988745be4211b6f48d3d339dc5c6 P 56036654e641486b99a44119fcff490f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df439d14be854ee5a671c9817319a4d4 in term 0.
I20260926 06:15:29.273895 10370 leader_election.cc:302] T ca4a988745be4211b6f48d3d339dc5c6 P df439d14be854ee5a671c9817319a4d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b54d024ec6964d07844331e5a81db31e, df439d14be854ee5a671c9817319a4d4; no voters:
I20260926 06:15:29.274049 10519 raft_consensus.cc:2781] T ca4a988745be4211b6f48d3d339dc5c6 P df439d14be854ee5a671c9817319a4d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:15:29.274096 10519 raft_consensus.cc:482] T ca4a988745be4211b6f48d3d339dc5c6 P df439d14be854ee5a671c9817319a4d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:15:29.274112 10519 raft_consensus.cc:3037] T ca4a988745be4211b6f48d3d339dc5c6 P df439d14be854ee5a671c9817319a4d4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:29.274837 10519 raft_consensus.cc:504] T ca4a988745be4211b6f48d3d339dc5c6 P df439d14be854ee5a671c9817319a4d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } } peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } }
I20260926 06:15:29.274945 10519 leader_election.cc:288] T ca4a988745be4211b6f48d3d339dc5c6 P df439d14be854ee5a671c9817319a4d4 [CANDIDATE]: Term 1 election: Requested vote from peers 56036654e641486b99a44119fcff490f (127.2.110.193:37843), b54d024ec6964d07844331e5a81db31e (127.2.110.194:36115)
I20260926 06:15:29.275195 10159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca4a988745be4211b6f48d3d339dc5c6" candidate_uuid: "df439d14be854ee5a671c9817319a4d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56036654e641486b99a44119fcff490f"
I20260926 06:15:29.275272 10159 raft_consensus.cc:3037] T ca4a988745be4211b6f48d3d339dc5c6 P 56036654e641486b99a44119fcff490f [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:29.275223 10298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca4a988745be4211b6f48d3d339dc5c6" candidate_uuid: "df439d14be854ee5a671c9817319a4d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b54d024ec6964d07844331e5a81db31e"
I20260926 06:15:29.275347 10298 raft_consensus.cc:3037] T ca4a988745be4211b6f48d3d339dc5c6 P b54d024ec6964d07844331e5a81db31e [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:29.275914 10159 raft_consensus.cc:2454] T ca4a988745be4211b6f48d3d339dc5c6 P 56036654e641486b99a44119fcff490f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df439d14be854ee5a671c9817319a4d4 in term 1.
I20260926 06:15:29.276113 10370 leader_election.cc:302] T ca4a988745be4211b6f48d3d339dc5c6 P df439d14be854ee5a671c9817319a4d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56036654e641486b99a44119fcff490f, df439d14be854ee5a671c9817319a4d4; no voters:
I20260926 06:15:29.276201 10298 raft_consensus.cc:2454] T ca4a988745be4211b6f48d3d339dc5c6 P b54d024ec6964d07844331e5a81db31e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df439d14be854ee5a671c9817319a4d4 in term 1.
I20260926 06:15:29.276240 10519 raft_consensus.cc:2781] T ca4a988745be4211b6f48d3d339dc5c6 P df439d14be854ee5a671c9817319a4d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:15:29.276309 10519 raft_consensus.cc:686] T ca4a988745be4211b6f48d3d339dc5c6 P df439d14be854ee5a671c9817319a4d4 [term 1 LEADER]: Becoming Leader. State: Replica: df439d14be854ee5a671c9817319a4d4, State: Running, Role: LEADER
I20260926 06:15:29.276364 10519 consensus_queue.cc:237] T ca4a988745be4211b6f48d3d339dc5c6 P df439d14be854ee5a671c9817319a4d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } } peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } }
I20260926 06:15:29.276732 10542 tablet_replica.cc:439] T ca4a988745be4211b6f48d3d339dc5c6 P df439d14be854ee5a671c9817319a4d4: TxnStatusTablet state changed. Reason: New leader df439d14be854ee5a671c9817319a4d4. Latest consensus state: current_term: 1 leader_uuid: "df439d14be854ee5a671c9817319a4d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } } peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } } }
I20260926 06:15:29.276853 10542 tablet_replica.cc:442] T ca4a988745be4211b6f48d3d339dc5c6 P df439d14be854ee5a671c9817319a4d4: This TxnStatusTablet replica's current role is: LEADER
I20260926 06:15:29.277055 10002 catalog_manager.cc:5495] T ca4a988745be4211b6f48d3d339dc5c6 P df439d14be854ee5a671c9817319a4d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to df439d14be854ee5a671c9817319a4d4 (127.2.110.195). New cstate: current_term: 1 leader_uuid: "df439d14be854ee5a671c9817319a4d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } health_report { overall_health: HEALTHY } } }
I20260926 06:15:29.277472 10554 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 06:15:29.277614 10554 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 06:15:29.306772 10549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:29.303254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.195:34709 (local address 127.0.0.1:41974)
0926 06:15:29.303339 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:29.303355 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:29.303419 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:29.303615 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:29.303616 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:29.303617 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 06:15:29.303669 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:29.303671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:29.304614 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:29.304615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:29.305519 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:29.305527 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:29.306140 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:29.306142 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:29.306214 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:29.306236 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 06:15:29.306589 (+ 353us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 06:15:29.306598 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 06:15:29.306641 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 06:15:29.306698 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 06:15:29.307029 10511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:29.303375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41974 (local address 127.2.110.195:34709)
0926 06:15:29.303449 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:29.303452 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:29.303462 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:15:29.303503 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:29.303504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:29.303526 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:29.303602 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:15:29.303720 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:29.304501 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:29.305666 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:29.305975 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:29.306069 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:29.306355 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:15:29.306510 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:15:29.306718 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:29.306761 (+ 43us) server_negotiation.cc:299] Negotiation successful
0926 06:15:29.306809 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 06:15:29.308553 10159 raft_consensus.cc:1261] T ca4a988745be4211b6f48d3d339dc5c6 P 56036654e641486b99a44119fcff490f [term 1 FOLLOWER]: Refusing update from remote peer df439d14be854ee5a671c9817319a4d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:15:29.308634 10298 raft_consensus.cc:1261] T ca4a988745be4211b6f48d3d339dc5c6 P b54d024ec6964d07844331e5a81db31e [term 1 FOLLOWER]: Refusing update from remote peer df439d14be854ee5a671c9817319a4d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:15:29.308851 10519 consensus_queue.cc:1035] T ca4a988745be4211b6f48d3d339dc5c6 P df439d14be854ee5a671c9817319a4d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:15:29.309154 10519 consensus_queue.cc:1035] T ca4a988745be4211b6f48d3d339dc5c6 P df439d14be854ee5a671c9817319a4d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:15:29.309538 10517 tablet_replica.cc:439] T ca4a988745be4211b6f48d3d339dc5c6 P 56036654e641486b99a44119fcff490f: TxnStatusTablet state changed. Reason: New leader df439d14be854ee5a671c9817319a4d4. Latest consensus state: current_term: 1 leader_uuid: "df439d14be854ee5a671c9817319a4d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } } peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } } }
I20260926 06:15:29.309803 10517 tablet_replica.cc:442] T ca4a988745be4211b6f48d3d339dc5c6 P 56036654e641486b99a44119fcff490f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:15:29.310161 10523 tablet_replica.cc:439] T ca4a988745be4211b6f48d3d339dc5c6 P b54d024ec6964d07844331e5a81db31e: TxnStatusTablet state changed. Reason: New leader df439d14be854ee5a671c9817319a4d4. Latest consensus state: current_term: 1 leader_uuid: "df439d14be854ee5a671c9817319a4d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } } peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } } }
I20260926 06:15:29.310289 10523 tablet_replica.cc:442] T ca4a988745be4211b6f48d3d339dc5c6 P b54d024ec6964d07844331e5a81db31e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:15:29.311057 10542 tablet_replica.cc:439] T ca4a988745be4211b6f48d3d339dc5c6 P df439d14be854ee5a671c9817319a4d4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "df439d14be854ee5a671c9817319a4d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } } peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } } }
I20260926 06:15:29.311185 10542 tablet_replica.cc:442] T ca4a988745be4211b6f48d3d339dc5c6 P df439d14be854ee5a671c9817319a4d4: This TxnStatusTablet replica's current role is: LEADER
I20260926 06:15:29.312438 10519 tablet_replica.cc:439] T ca4a988745be4211b6f48d3d339dc5c6 P df439d14be854ee5a671c9817319a4d4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "df439d14be854ee5a671c9817319a4d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } } peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } } }
I20260926 06:15:29.312531 10519 tablet_replica.cc:442] T ca4a988745be4211b6f48d3d339dc5c6 P df439d14be854ee5a671c9817319a4d4: This TxnStatusTablet replica's current role is: LEADER
I20260926 06:15:29.312700 10523 tablet_replica.cc:439] T ca4a988745be4211b6f48d3d339dc5c6 P b54d024ec6964d07844331e5a81db31e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "df439d14be854ee5a671c9817319a4d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } } peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } } }
I20260926 06:15:29.312799 10523 tablet_replica.cc:442] T ca4a988745be4211b6f48d3d339dc5c6 P b54d024ec6964d07844331e5a81db31e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:15:29.313172 10517 tablet_replica.cc:439] T ca4a988745be4211b6f48d3d339dc5c6 P 56036654e641486b99a44119fcff490f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "df439d14be854ee5a671c9817319a4d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } } peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } } }
I20260926 06:15:29.313594 10517 tablet_replica.cc:442] T ca4a988745be4211b6f48d3d339dc5c6 P 56036654e641486b99a44119fcff490f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:15:29.316383 10511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:29.313309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41986 (local address 127.2.110.195:34709)
0926 06:15:29.313490 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:29.313492 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:29.313501 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:15:29.313598 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:29.313599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:29.313614 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:29.313677 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:15:29.313889 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:29.314675 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:29.315462 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:29.315799 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:29.315885 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:29.316054 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:15:29.316092 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:15:29.316153 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:29.316270 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 06:15:29.316294 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
I20260926 06:15:29.322290 10562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:29.319213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.195:34709 (local address 127.2.110.195:51539)
0926 06:15:29.319408 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:29.319418 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:29.319477 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:29.319603 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:29.319604 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:29.319605 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 06:15:29.319660 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:29.319661 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:29.320596 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:29.320597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:29.321187 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:29.321189 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:29.321789 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:29.321791 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:29.321861 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:29.321879 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 06:15:29.322087 (+ 208us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 06:15:29.322088 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 06:15:29.322134 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 06:15:29.322174 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":133,"threads_started":1}
I20260926 06:15:29.322294 10511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:29.319286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:51539 (local address 127.2.110.195:34709)
0926 06:15:29.319334 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:29.319337 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:29.319489 (+ 152us) server_negotiation.cc:407] Connection header received
0926 06:15:29.319514 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:29.319515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:29.319525 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:29.319587 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:15:29.319720 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:29.320512 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:29.321284 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:29.321626 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:29.321705 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:29.321981 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:15:29.322017 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:15:29.322074 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:29.322152 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 06:15:29.322171 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 06:15:29.325034 10159 raft_consensus.cc:1261] T 413b877177164ab293886930864c9329 P 56036654e641486b99a44119fcff490f [term 1 FOLLOWER]: Refusing update from remote peer df439d14be854ee5a671c9817319a4d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:15:29.325081 10298 raft_consensus.cc:1261] T 413b877177164ab293886930864c9329 P b54d024ec6964d07844331e5a81db31e [term 1 FOLLOWER]: Refusing update from remote peer df439d14be854ee5a671c9817319a4d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:15:29.325251 10542 consensus_queue.cc:1035] T 413b877177164ab293886930864c9329 P df439d14be854ee5a671c9817319a4d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:15:29.325402 10559 consensus_queue.cc:1035] T 413b877177164ab293886930864c9329 P df439d14be854ee5a671c9817319a4d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:15:29.339468 10000 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:15:29.339700 10000 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:15:29.340523 10000 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 24536bc278a44d40bfcee7eb68808bf6: Sending DeleteTablet for 3 replicas of tablet 413b877177164ab293886930864c9329
I20260926 06:15:29.340816 10139 tablet_service.cc:1514] Processing DeleteTablet for tablet 413b877177164ab293886930864c9329 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:15:29 UTC) from {username='oryx', principal='oryx/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:34980
I20260926 06:15:29.340822 10279 tablet_service.cc:1514] Processing DeleteTablet for tablet 413b877177164ab293886930864c9329 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:15:29 UTC) from {username='oryx', principal='oryx/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:33412
I20260926 06:15:29.340917 10535 tablet_replica.cc:330] T 413b877177164ab293886930864c9329 P b54d024ec6964d07844331e5a81db31e: stopping tablet replica
I20260926 06:15:29.340937 10536 tablet_replica.cc:330] T 413b877177164ab293886930864c9329 P 56036654e641486b99a44119fcff490f: stopping tablet replica
I20260926 06:15:29.340971 10535 raft_consensus.cc:2229] T 413b877177164ab293886930864c9329 P b54d024ec6964d07844331e5a81db31e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:15:29.340982 10536 raft_consensus.cc:2229] T 413b877177164ab293886930864c9329 P 56036654e641486b99a44119fcff490f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:15:29.341009 10535 raft_consensus.cc:2258] T 413b877177164ab293886930864c9329 P b54d024ec6964d07844331e5a81db31e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:15:29.341017 10536 raft_consensus.cc:2258] T 413b877177164ab293886930864c9329 P 56036654e641486b99a44119fcff490f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:15:29.341189 10419 tablet_service.cc:1514] Processing DeleteTablet for tablet 413b877177164ab293886930864c9329 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:15:29 UTC) from {username='oryx', principal='oryx/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:41970
I20260926 06:15:29.341305 10537 tablet_replica.cc:330] T 413b877177164ab293886930864c9329 P df439d14be854ee5a671c9817319a4d4: stopping tablet replica
I20260926 06:15:29.341382 10537 raft_consensus.cc:2229] T 413b877177164ab293886930864c9329 P df439d14be854ee5a671c9817319a4d4 [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:15:29.341396 10535 ts_tablet_manager.cc:1905] T 413b877177164ab293886930864c9329 P b54d024ec6964d07844331e5a81db31e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:15:29.341481 10537 raft_consensus.cc:2258] T 413b877177164ab293886930864c9329 P df439d14be854ee5a671c9817319a4d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:15:29.341609 10536 ts_tablet_manager.cc:1905] T 413b877177164ab293886930864c9329 P 56036654e641486b99a44119fcff490f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:15:29.341746 10537 ts_tablet_manager.cc:1905] T 413b877177164ab293886930864c9329 P df439d14be854ee5a671c9817319a4d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:15:29.343326 10536 ts_tablet_manager.cc:1918] T 413b877177164ab293886930864c9329 P 56036654e641486b99a44119fcff490f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 06:15:29.343380 10537 ts_tablet_manager.cc:1918] T 413b877177164ab293886930864c9329 P df439d14be854ee5a671c9817319a4d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 06:15:29.343395 10536 log.cc:1196] T 413b877177164ab293886930864c9329 P 56036654e641486b99a44119fcff490f: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-0/wal/wals/413b877177164ab293886930864c9329
I20260926 06:15:29.343428 10537 log.cc:1196] T 413b877177164ab293886930864c9329 P df439d14be854ee5a671c9817319a4d4: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-2/wal/wals/413b877177164ab293886930864c9329
I20260926 06:15:29.343636 10536 ts_tablet_manager.cc:1939] T 413b877177164ab293886930864c9329 P 56036654e641486b99a44119fcff490f: Deleting consensus metadata
I20260926 06:15:29.343636 10537 ts_tablet_manager.cc:1939] T 413b877177164ab293886930864c9329 P df439d14be854ee5a671c9817319a4d4: Deleting consensus metadata
I20260926 06:15:29.343950 10535 ts_tablet_manager.cc:1918] T 413b877177164ab293886930864c9329 P b54d024ec6964d07844331e5a81db31e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 06:15:29.343998 10535 log.cc:1196] T 413b877177164ab293886930864c9329 P b54d024ec6964d07844331e5a81db31e: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-1/wal/wals/413b877177164ab293886930864c9329
I20260926 06:15:29.344084 9987 catalog_manager.cc:4841] TS df439d14be854ee5a671c9817319a4d4 (127.2.110.195:34709): tablet 413b877177164ab293886930864c9329 (table test-table [id=27b0ca7b1dea434bad28f27f3d0ffd31]) successfully deleted
I20260926 06:15:29.344174 9986 catalog_manager.cc:4841] TS 56036654e641486b99a44119fcff490f (127.2.110.193:37843): tablet 413b877177164ab293886930864c9329 (table test-table [id=27b0ca7b1dea434bad28f27f3d0ffd31]) successfully deleted
I20260926 06:15:29.344199 10535 ts_tablet_manager.cc:1939] T 413b877177164ab293886930864c9329 P b54d024ec6964d07844331e5a81db31e: Deleting consensus metadata
I20260926 06:15:29.344552 9986 catalog_manager.cc:4841] TS b54d024ec6964d07844331e5a81db31e (127.2.110.194:36115): tablet 413b877177164ab293886930864c9329 (table test-table [id=27b0ca7b1dea434bad28f27f3d0ffd31]) successfully deleted
Sep 26 06:15:29 dist-test-slave-hndd krb5kdc[9957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 06:15:29 dist-test-slave-hndd krb5kdc[9957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:15:29 dist-test-slave-hndd krb5kdc[9957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 06:15:29.368763 10052 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:29.362027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53656 (local address 127.2.110.254:46239)
0926 06:15:29.362104 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:29.362107 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:29.362260 (+ 153us) server_negotiation.cc:407] Connection header received
0926 06:15:29.362337 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:29.362338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:29.362348 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:29.362400 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:29.363148 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:29.363728 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:29.364796 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:29.364989 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:29.365047 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:29.368672 (+ 3625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53656: BlockingRecv error: recv got EOF from 127.0.0.1:53656 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 26 06:15:29 dist-test-slave-hndd krb5kdc[9957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM
I20260926 06:15:29.378795 10052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:29.369732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53660 (local address 127.2.110.254:46239)
0926 06:15:29.369788 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:29.369790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:29.369985 (+ 195us) server_negotiation.cc:407] Connection header received
0926 06:15:29.370132 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:29.370133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:29.370147 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:29.370212 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:29.370820 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:29.371293 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:29.372327 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:29.372513 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:29.372571 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:29.375658 (+ 3087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:29.375660 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:29.375660 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:29.375669 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:29.377271 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:29.377949 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:29.377950 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:29.377951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:29.378020 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:29.378234 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:29.378234 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:29.378235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:29.378370 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:29.378484 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:29.378601 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 06:15:29.378689 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 06:15:29.381085 10000 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53660:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:15:29.381453 10000 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:15:29.382928 10279 tablet_service.cc:1467] Processing CreateTablet for tablet 48d137241db64df78a3f2a1a2c85bf86 (DEFAULT_TABLE table=test-table [id=ebb77314914d4dd4ae75d7c75e85331d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:15:29.382928 10419 tablet_service.cc:1467] Processing CreateTablet for tablet 48d137241db64df78a3f2a1a2c85bf86 (DEFAULT_TABLE table=test-table [id=ebb77314914d4dd4ae75d7c75e85331d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:15:29.382937 10139 tablet_service.cc:1467] Processing CreateTablet for tablet 48d137241db64df78a3f2a1a2c85bf86 (DEFAULT_TABLE table=test-table [id=ebb77314914d4dd4ae75d7c75e85331d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:15:29.383075 10139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48d137241db64df78a3f2a1a2c85bf86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:29.383073 10279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48d137241db64df78a3f2a1a2c85bf86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:29.383074 10419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48d137241db64df78a3f2a1a2c85bf86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:29.384481 10518 tablet_bootstrap.cc:492] T 48d137241db64df78a3f2a1a2c85bf86 P b54d024ec6964d07844331e5a81db31e: Bootstrap starting.
I20260926 06:15:29.384541 10513 tablet_bootstrap.cc:492] T 48d137241db64df78a3f2a1a2c85bf86 P 56036654e641486b99a44119fcff490f: Bootstrap starting.
I20260926 06:15:29.384877 10518 tablet_bootstrap.cc:654] T 48d137241db64df78a3f2a1a2c85bf86 P b54d024ec6964d07844331e5a81db31e: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:29.384941 10513 tablet_bootstrap.cc:654] T 48d137241db64df78a3f2a1a2c85bf86 P 56036654e641486b99a44119fcff490f: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:29.385422 10514 tablet_bootstrap.cc:492] T 48d137241db64df78a3f2a1a2c85bf86 P df439d14be854ee5a671c9817319a4d4: Bootstrap starting.
I20260926 06:15:29.385464 10518 tablet_bootstrap.cc:492] T 48d137241db64df78a3f2a1a2c85bf86 P b54d024ec6964d07844331e5a81db31e: No bootstrap required, opened a new log
I20260926 06:15:29.385499 10518 ts_tablet_manager.cc:1397] T 48d137241db64df78a3f2a1a2c85bf86 P b54d024ec6964d07844331e5a81db31e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:29.385614 10518 raft_consensus.cc:348] T 48d137241db64df78a3f2a1a2c85bf86 P b54d024ec6964d07844331e5a81db31e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } } peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } }
I20260926 06:15:29.385684 10518 raft_consensus.cc:374] T 48d137241db64df78a3f2a1a2c85bf86 P b54d024ec6964d07844331e5a81db31e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:29.385703 10518 raft_consensus.cc:729] T 48d137241db64df78a3f2a1a2c85bf86 P b54d024ec6964d07844331e5a81db31e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b54d024ec6964d07844331e5a81db31e, State: Initialized, Role: FOLLOWER
I20260926 06:15:29.385767 10518 consensus_queue.cc:260] T 48d137241db64df78a3f2a1a2c85bf86 P b54d024ec6964d07844331e5a81db31e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } } peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } }
I20260926 06:15:29.385812 10514 tablet_bootstrap.cc:654] T 48d137241db64df78a3f2a1a2c85bf86 P df439d14be854ee5a671c9817319a4d4: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:29.385891 10518 ts_tablet_manager.cc:1428] T 48d137241db64df78a3f2a1a2c85bf86 P b54d024ec6964d07844331e5a81db31e: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 06:15:29.386613 10514 tablet_bootstrap.cc:492] T 48d137241db64df78a3f2a1a2c85bf86 P df439d14be854ee5a671c9817319a4d4: No bootstrap required, opened a new log
I20260926 06:15:29.386672 10514 ts_tablet_manager.cc:1397] T 48d137241db64df78a3f2a1a2c85bf86 P df439d14be854ee5a671c9817319a4d4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:29.386853 10514 raft_consensus.cc:348] T 48d137241db64df78a3f2a1a2c85bf86 P df439d14be854ee5a671c9817319a4d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } } peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } }
I20260926 06:15:29.386929 10514 raft_consensus.cc:374] T 48d137241db64df78a3f2a1a2c85bf86 P df439d14be854ee5a671c9817319a4d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:29.386950 10514 raft_consensus.cc:729] T 48d137241db64df78a3f2a1a2c85bf86 P df439d14be854ee5a671c9817319a4d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df439d14be854ee5a671c9817319a4d4, State: Initialized, Role: FOLLOWER
I20260926 06:15:29.386999 10514 consensus_queue.cc:260] T 48d137241db64df78a3f2a1a2c85bf86 P df439d14be854ee5a671c9817319a4d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } } peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } }
I20260926 06:15:29.387084 10514 ts_tablet_manager.cc:1428] T 48d137241db64df78a3f2a1a2c85bf86 P df439d14be854ee5a671c9817319a4d4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:29.387317 10513 tablet_bootstrap.cc:492] T 48d137241db64df78a3f2a1a2c85bf86 P 56036654e641486b99a44119fcff490f: No bootstrap required, opened a new log
I20260926 06:15:29.387369 10513 ts_tablet_manager.cc:1397] T 48d137241db64df78a3f2a1a2c85bf86 P 56036654e641486b99a44119fcff490f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 06:15:29.387517 10513 raft_consensus.cc:348] T 48d137241db64df78a3f2a1a2c85bf86 P 56036654e641486b99a44119fcff490f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } } peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } }
I20260926 06:15:29.387589 10513 raft_consensus.cc:374] T 48d137241db64df78a3f2a1a2c85bf86 P 56036654e641486b99a44119fcff490f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:29.387610 10513 raft_consensus.cc:729] T 48d137241db64df78a3f2a1a2c85bf86 P 56036654e641486b99a44119fcff490f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56036654e641486b99a44119fcff490f, State: Initialized, Role: FOLLOWER
I20260926 06:15:29.387655 10513 consensus_queue.cc:260] T 48d137241db64df78a3f2a1a2c85bf86 P 56036654e641486b99a44119fcff490f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } } peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } }
I20260926 06:15:29.387742 10513 ts_tablet_manager.cc:1428] T 48d137241db64df78a3f2a1a2c85bf86 P 56036654e641486b99a44119fcff490f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:29.622466 10517 raft_consensus.cc:482] T 48d137241db64df78a3f2a1a2c85bf86 P 56036654e641486b99a44119fcff490f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:15:29.622602 10517 raft_consensus.cc:504] T 48d137241db64df78a3f2a1a2c85bf86 P 56036654e641486b99a44119fcff490f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } } peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } }
I20260926 06:15:29.622982 10517 leader_election.cc:288] T 48d137241db64df78a3f2a1a2c85bf86 P 56036654e641486b99a44119fcff490f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b54d024ec6964d07844331e5a81db31e (127.2.110.194:36115), df439d14be854ee5a671c9817319a4d4 (127.2.110.195:34709)
I20260926 06:15:29.626214 10511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:29.623274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:52967 (local address 127.2.110.195:34709)
0926 06:15:29.623332 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:29.623335 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:29.623479 (+ 144us) server_negotiation.cc:407] Connection header received
0926 06:15:29.623553 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:29.623554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:29.623564 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:29.623634 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:15:29.623791 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:29.624273 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:29.625475 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:29.625855 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:29.625938 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:29.625949 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:29.626093 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 06:15:29.626128 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 06:15:29.626258 10584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:29.623370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:48175 (local address 127.2.110.194:36115)
0926 06:15:29.623491 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:29.623492 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:29.623500 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:15:29.623531 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:29.623531 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:29.623542 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:29.623634 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:15:29.623812 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:29.624284 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:29.625446 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:29.625903 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:29.625987 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:29.625996 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:29.626138 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 06:15:29.626182 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":67,"threads_started":1}
I20260926 06:15:29.626264 10582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:29.623216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.195:34709 (local address 127.2.110.193:52967)
0926 06:15:29.623371 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:29.623384 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:29.623487 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:29.623639 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:29.623640 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:29.623643 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:15:29.623717 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:29.623718 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:29.624372 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:29.624373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:29.625378 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:29.625381 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:29.625948 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:29.625949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:29.626011 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:29.626018 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 06:15:29.626090 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 06:15:29.626132 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":52,"thread_start_us":65,"threads_started":1}
I20260926 06:15:29.626295 10583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:29.623270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.194:36115 (local address 127.2.110.193:48175)
0926 06:15:29.623384 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:29.623397 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:29.623459 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:29.623642 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:29.623644 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:29.623646 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:15:29.623723 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:29.623725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:29.624374 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:29.624375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:29.625325 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:29.625327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:29.625985 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:29.625986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:29.626056 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:29.626060 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:15:29.626120 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 06:15:29.626165 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":7,"thread_start_us":76,"threads_started":1}
I20260926 06:15:29.626669 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48d137241db64df78a3f2a1a2c85bf86" candidate_uuid: "56036654e641486b99a44119fcff490f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df439d14be854ee5a671c9817319a4d4" is_pre_election: true
I20260926 06:15:29.626798 10438 raft_consensus.cc:2454] T 48d137241db64df78a3f2a1a2c85bf86 P df439d14be854ee5a671c9817319a4d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56036654e641486b99a44119fcff490f in term 0.
I20260926 06:15:29.627064 10091 leader_election.cc:302] T 48d137241db64df78a3f2a1a2c85bf86 P 56036654e641486b99a44119fcff490f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56036654e641486b99a44119fcff490f, df439d14be854ee5a671c9817319a4d4; no voters:
I20260926 06:15:29.627100 10298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48d137241db64df78a3f2a1a2c85bf86" candidate_uuid: "56036654e641486b99a44119fcff490f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b54d024ec6964d07844331e5a81db31e" is_pre_election: true
I20260926 06:15:29.627202 10298 raft_consensus.cc:2454] T 48d137241db64df78a3f2a1a2c85bf86 P b54d024ec6964d07844331e5a81db31e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56036654e641486b99a44119fcff490f in term 0.
I20260926 06:15:29.627209 10517 raft_consensus.cc:2781] T 48d137241db64df78a3f2a1a2c85bf86 P 56036654e641486b99a44119fcff490f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:15:29.627250 10517 raft_consensus.cc:482] T 48d137241db64df78a3f2a1a2c85bf86 P 56036654e641486b99a44119fcff490f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:15:29.627274 10517 raft_consensus.cc:3037] T 48d137241db64df78a3f2a1a2c85bf86 P 56036654e641486b99a44119fcff490f [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:29.628038 10517 raft_consensus.cc:504] T 48d137241db64df78a3f2a1a2c85bf86 P 56036654e641486b99a44119fcff490f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } } peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } }
I20260926 06:15:29.628199 10517 leader_election.cc:288] T 48d137241db64df78a3f2a1a2c85bf86 P 56036654e641486b99a44119fcff490f [CANDIDATE]: Term 1 election: Requested vote from peers b54d024ec6964d07844331e5a81db31e (127.2.110.194:36115), df439d14be854ee5a671c9817319a4d4 (127.2.110.195:34709)
I20260926 06:15:29.628335 10298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48d137241db64df78a3f2a1a2c85bf86" candidate_uuid: "56036654e641486b99a44119fcff490f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b54d024ec6964d07844331e5a81db31e"
I20260926 06:15:29.628418 10298 raft_consensus.cc:3037] T 48d137241db64df78a3f2a1a2c85bf86 P b54d024ec6964d07844331e5a81db31e [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:29.628407 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48d137241db64df78a3f2a1a2c85bf86" candidate_uuid: "56036654e641486b99a44119fcff490f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df439d14be854ee5a671c9817319a4d4"
I20260926 06:15:29.628468 10438 raft_consensus.cc:3037] T 48d137241db64df78a3f2a1a2c85bf86 P df439d14be854ee5a671c9817319a4d4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:29.629793 10438 raft_consensus.cc:2454] T 48d137241db64df78a3f2a1a2c85bf86 P df439d14be854ee5a671c9817319a4d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56036654e641486b99a44119fcff490f in term 1.
I20260926 06:15:29.629793 10298 raft_consensus.cc:2454] T 48d137241db64df78a3f2a1a2c85bf86 P b54d024ec6964d07844331e5a81db31e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56036654e641486b99a44119fcff490f in term 1.
I20260926 06:15:29.630158 10091 leader_election.cc:302] T 48d137241db64df78a3f2a1a2c85bf86 P 56036654e641486b99a44119fcff490f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56036654e641486b99a44119fcff490f, df439d14be854ee5a671c9817319a4d4; no voters:
I20260926 06:15:29.630275 10517 raft_consensus.cc:2781] T 48d137241db64df78a3f2a1a2c85bf86 P 56036654e641486b99a44119fcff490f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:15:29.630487 10517 raft_consensus.cc:686] T 48d137241db64df78a3f2a1a2c85bf86 P 56036654e641486b99a44119fcff490f [term 1 LEADER]: Becoming Leader. State: Replica: 56036654e641486b99a44119fcff490f, State: Running, Role: LEADER
I20260926 06:15:29.630602 10517 consensus_queue.cc:237] T 48d137241db64df78a3f2a1a2c85bf86 P 56036654e641486b99a44119fcff490f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } } peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } }
I20260926 06:15:29.631903 10002 catalog_manager.cc:5495] T 48d137241db64df78a3f2a1a2c85bf86 P 56036654e641486b99a44119fcff490f reported cstate change: term changed from 0 to 1, leader changed from <none> to 56036654e641486b99a44119fcff490f (127.2.110.193). New cstate: current_term: 1 leader_uuid: "56036654e641486b99a44119fcff490f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } health_report { overall_health: UNKNOWN } } }
I20260926 06:15:29.675253 10588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:29.672283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34990 (local address 127.2.110.193:37843)
0926 06:15:29.672473 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:29.672475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:29.672481 (+ 6us) server_negotiation.cc:407] Connection header received
0926 06:15:29.672521 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:29.672521 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:29.672541 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:29.672643 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:15:29.672818 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:29.673487 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:29.674354 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:29.674587 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:29.674669 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:29.674851 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:15:29.674973 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:15:29.675062 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:29.675101 (+ 39us) server_negotiation.cc:299] Negotiation successful
0926 06:15:29.675126 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":92,"threads_started":1}
I20260926 06:15:29.676301 10438 raft_consensus.cc:1261] T 48d137241db64df78a3f2a1a2c85bf86 P df439d14be854ee5a671c9817319a4d4 [term 1 FOLLOWER]: Refusing update from remote peer 56036654e641486b99a44119fcff490f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:15:29.676309 10298 raft_consensus.cc:1261] T 48d137241db64df78a3f2a1a2c85bf86 P b54d024ec6964d07844331e5a81db31e [term 1 FOLLOWER]: Refusing update from remote peer 56036654e641486b99a44119fcff490f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:15:29.676561 10585 consensus_queue.cc:1035] T 48d137241db64df78a3f2a1a2c85bf86 P 56036654e641486b99a44119fcff490f [LEADER]: Connected to new peer: Peer: permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:15:29.676715 10517 consensus_queue.cc:1035] T 48d137241db64df78a3f2a1a2c85bf86 P 56036654e641486b99a44119fcff490f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:15:29.679952 10000 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:15:29.680073 10000 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:15:29.680853 10000 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 24536bc278a44d40bfcee7eb68808bf6: Sending DeleteTablet for 3 replicas of tablet 48d137241db64df78a3f2a1a2c85bf86
I20260926 06:15:29.681121 10419 tablet_service.cc:1514] Processing DeleteTablet for tablet 48d137241db64df78a3f2a1a2c85bf86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:15:29 UTC) from {username='oryx', principal='oryx/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:41970
I20260926 06:15:29.681236 10537 tablet_replica.cc:330] T 48d137241db64df78a3f2a1a2c85bf86 P df439d14be854ee5a671c9817319a4d4: stopping tablet replica
I20260926 06:15:29.681255 10279 tablet_service.cc:1514] Processing DeleteTablet for tablet 48d137241db64df78a3f2a1a2c85bf86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:15:29 UTC) from {username='oryx', principal='oryx/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:33412
I20260926 06:15:29.681305 10537 raft_consensus.cc:2229] T 48d137241db64df78a3f2a1a2c85bf86 P df439d14be854ee5a671c9817319a4d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:15:29.681361 10139 tablet_service.cc:1514] Processing DeleteTablet for tablet 48d137241db64df78a3f2a1a2c85bf86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:15:29 UTC) from {username='oryx', principal='oryx/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:34980
I20260926 06:15:29.681388 10537 raft_consensus.cc:2258] T 48d137241db64df78a3f2a1a2c85bf86 P df439d14be854ee5a671c9817319a4d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:15:29.681452 10535 tablet_replica.cc:330] T 48d137241db64df78a3f2a1a2c85bf86 P b54d024ec6964d07844331e5a81db31e: stopping tablet replica
I20260926 06:15:29.681504 10535 raft_consensus.cc:2229] T 48d137241db64df78a3f2a1a2c85bf86 P b54d024ec6964d07844331e5a81db31e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:15:29.681547 10535 raft_consensus.cc:2258] T 48d137241db64df78a3f2a1a2c85bf86 P b54d024ec6964d07844331e5a81db31e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:15:29.681620 10536 tablet_replica.cc:330] T 48d137241db64df78a3f2a1a2c85bf86 P 56036654e641486b99a44119fcff490f: stopping tablet replica
I20260926 06:15:29.681674 10536 raft_consensus.cc:2229] T 48d137241db64df78a3f2a1a2c85bf86 P 56036654e641486b99a44119fcff490f [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:15:29.681762 10536 raft_consensus.cc:2258] T 48d137241db64df78a3f2a1a2c85bf86 P 56036654e641486b99a44119fcff490f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:15:29.682076 10537 ts_tablet_manager.cc:1905] T 48d137241db64df78a3f2a1a2c85bf86 P df439d14be854ee5a671c9817319a4d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:15:29.682145 10535 ts_tablet_manager.cc:1905] T 48d137241db64df78a3f2a1a2c85bf86 P b54d024ec6964d07844331e5a81db31e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:15:29.683670 10000 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53660:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 06:15:29.683786 10536 ts_tablet_manager.cc:1905] T 48d137241db64df78a3f2a1a2c85bf86 P 56036654e641486b99a44119fcff490f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 06:15:29.683823 10000 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:15:29.684038 10535 ts_tablet_manager.cc:1918] T 48d137241db64df78a3f2a1a2c85bf86 P b54d024ec6964d07844331e5a81db31e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:15:29.684111 10535 log.cc:1196] T 48d137241db64df78a3f2a1a2c85bf86 P b54d024ec6964d07844331e5a81db31e: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-1/wal/wals/48d137241db64df78a3f2a1a2c85bf86
I20260926 06:15:29.684352 10535 ts_tablet_manager.cc:1939] T 48d137241db64df78a3f2a1a2c85bf86 P b54d024ec6964d07844331e5a81db31e: Deleting consensus metadata
I20260926 06:15:29.684655 9986 catalog_manager.cc:4841] TS b54d024ec6964d07844331e5a81db31e (127.2.110.194:36115): tablet 48d137241db64df78a3f2a1a2c85bf86 (table test-table [id=ebb77314914d4dd4ae75d7c75e85331d]) successfully deleted
I20260926 06:15:29.684779 10537 ts_tablet_manager.cc:1918] T 48d137241db64df78a3f2a1a2c85bf86 P df439d14be854ee5a671c9817319a4d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:15:29.684829 10537 log.cc:1196] T 48d137241db64df78a3f2a1a2c85bf86 P df439d14be854ee5a671c9817319a4d4: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-2/wal/wals/48d137241db64df78a3f2a1a2c85bf86
I20260926 06:15:29.685062 10537 ts_tablet_manager.cc:1939] T 48d137241db64df78a3f2a1a2c85bf86 P df439d14be854ee5a671c9817319a4d4: Deleting consensus metadata
I20260926 06:15:29.685245 10536 ts_tablet_manager.cc:1918] T 48d137241db64df78a3f2a1a2c85bf86 P 56036654e641486b99a44119fcff490f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:15:29.685294 10536 log.cc:1196] T 48d137241db64df78a3f2a1a2c85bf86 P 56036654e641486b99a44119fcff490f: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-0/wal/wals/48d137241db64df78a3f2a1a2c85bf86
I20260926 06:15:29.685375 10419 tablet_service.cc:1467] Processing CreateTablet for tablet 9546fd87f6e148e492af92b357f65fc1 (DEFAULT_TABLE table=test-table [id=a74c161978604e30b61457185bbd0b04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:15:29.685469 10419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9546fd87f6e148e492af92b357f65fc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:29.685534 10536 ts_tablet_manager.cc:1939] T 48d137241db64df78a3f2a1a2c85bf86 P 56036654e641486b99a44119fcff490f: Deleting consensus metadata
I20260926 06:15:29.686631 10514 tablet_bootstrap.cc:492] T 9546fd87f6e148e492af92b357f65fc1 P df439d14be854ee5a671c9817319a4d4: Bootstrap starting.
I20260926 06:15:29.686669 9987 catalog_manager.cc:4841] TS df439d14be854ee5a671c9817319a4d4 (127.2.110.195:34709): tablet 48d137241db64df78a3f2a1a2c85bf86 (table test-table [id=ebb77314914d4dd4ae75d7c75e85331d]) successfully deleted
I20260926 06:15:29.686954 10514 tablet_bootstrap.cc:654] T 9546fd87f6e148e492af92b357f65fc1 P df439d14be854ee5a671c9817319a4d4: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:29.686944 9986 catalog_manager.cc:4841] TS 56036654e641486b99a44119fcff490f (127.2.110.193:37843): tablet 48d137241db64df78a3f2a1a2c85bf86 (table test-table [id=ebb77314914d4dd4ae75d7c75e85331d]) successfully deleted
I20260926 06:15:29.687050 10139 tablet_service.cc:1467] Processing CreateTablet for tablet 9546fd87f6e148e492af92b357f65fc1 (DEFAULT_TABLE table=test-table [id=a74c161978604e30b61457185bbd0b04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:15:29.687151 10139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9546fd87f6e148e492af92b357f65fc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:29.687120 10279 tablet_service.cc:1467] Processing CreateTablet for tablet 9546fd87f6e148e492af92b357f65fc1 (DEFAULT_TABLE table=test-table [id=a74c161978604e30b61457185bbd0b04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:15:29.687237 10279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9546fd87f6e148e492af92b357f65fc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:29.688371 10518 tablet_bootstrap.cc:492] T 9546fd87f6e148e492af92b357f65fc1 P b54d024ec6964d07844331e5a81db31e: Bootstrap starting.
I20260926 06:15:29.688736 10518 tablet_bootstrap.cc:654] T 9546fd87f6e148e492af92b357f65fc1 P b54d024ec6964d07844331e5a81db31e: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:29.689064 10513 tablet_bootstrap.cc:492] T 9546fd87f6e148e492af92b357f65fc1 P 56036654e641486b99a44119fcff490f: Bootstrap starting.
I20260926 06:15:29.689502 10513 tablet_bootstrap.cc:654] T 9546fd87f6e148e492af92b357f65fc1 P 56036654e641486b99a44119fcff490f: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:29.689945 10518 tablet_bootstrap.cc:492] T 9546fd87f6e148e492af92b357f65fc1 P b54d024ec6964d07844331e5a81db31e: No bootstrap required, opened a new log
I20260926 06:15:29.690006 10518 ts_tablet_manager.cc:1397] T 9546fd87f6e148e492af92b357f65fc1 P b54d024ec6964d07844331e5a81db31e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 06:15:29.690146 10518 raft_consensus.cc:348] T 9546fd87f6e148e492af92b357f65fc1 P b54d024ec6964d07844331e5a81db31e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } } peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } }
I20260926 06:15:29.690234 10518 raft_consensus.cc:374] T 9546fd87f6e148e492af92b357f65fc1 P b54d024ec6964d07844331e5a81db31e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:29.690260 10518 raft_consensus.cc:729] T 9546fd87f6e148e492af92b357f65fc1 P b54d024ec6964d07844331e5a81db31e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b54d024ec6964d07844331e5a81db31e, State: Initialized, Role: FOLLOWER
I20260926 06:15:29.690310 10518 consensus_queue.cc:260] T 9546fd87f6e148e492af92b357f65fc1 P b54d024ec6964d07844331e5a81db31e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } } peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } }
I20260926 06:15:29.690408 10518 ts_tablet_manager.cc:1428] T 9546fd87f6e148e492af92b357f65fc1 P b54d024ec6964d07844331e5a81db31e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:29.690706 10513 tablet_bootstrap.cc:492] T 9546fd87f6e148e492af92b357f65fc1 P 56036654e641486b99a44119fcff490f: No bootstrap required, opened a new log
I20260926 06:15:29.690771 10513 ts_tablet_manager.cc:1397] T 9546fd87f6e148e492af92b357f65fc1 P 56036654e641486b99a44119fcff490f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 06:15:29.690940 10513 raft_consensus.cc:348] T 9546fd87f6e148e492af92b357f65fc1 P 56036654e641486b99a44119fcff490f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } } peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } }
I20260926 06:15:29.691021 10513 raft_consensus.cc:374] T 9546fd87f6e148e492af92b357f65fc1 P 56036654e641486b99a44119fcff490f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:29.691051 10513 raft_consensus.cc:729] T 9546fd87f6e148e492af92b357f65fc1 P 56036654e641486b99a44119fcff490f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56036654e641486b99a44119fcff490f, State: Initialized, Role: FOLLOWER
I20260926 06:15:29.691097 10513 consensus_queue.cc:260] T 9546fd87f6e148e492af92b357f65fc1 P 56036654e641486b99a44119fcff490f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } } peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } }
I20260926 06:15:29.691191 10513 ts_tablet_manager.cc:1428] T 9546fd87f6e148e492af92b357f65fc1 P 56036654e641486b99a44119fcff490f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 06:15:29.697948 10514 tablet_bootstrap.cc:492] T 9546fd87f6e148e492af92b357f65fc1 P df439d14be854ee5a671c9817319a4d4: No bootstrap required, opened a new log
I20260926 06:15:29.698035 10514 ts_tablet_manager.cc:1397] T 9546fd87f6e148e492af92b357f65fc1 P df439d14be854ee5a671c9817319a4d4: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20260926 06:15:29.698177 10514 raft_consensus.cc:348] T 9546fd87f6e148e492af92b357f65fc1 P df439d14be854ee5a671c9817319a4d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } } peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } }
I20260926 06:15:29.698273 10514 raft_consensus.cc:374] T 9546fd87f6e148e492af92b357f65fc1 P df439d14be854ee5a671c9817319a4d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:29.698305 10514 raft_consensus.cc:729] T 9546fd87f6e148e492af92b357f65fc1 P df439d14be854ee5a671c9817319a4d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df439d14be854ee5a671c9817319a4d4, State: Initialized, Role: FOLLOWER
I20260926 06:15:29.698360 10514 consensus_queue.cc:260] T 9546fd87f6e148e492af92b357f65fc1 P df439d14be854ee5a671c9817319a4d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } } peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } }
I20260926 06:15:29.698464 10514 ts_tablet_manager.cc:1428] T 9546fd87f6e148e492af92b357f65fc1 P df439d14be854ee5a671c9817319a4d4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:29.782784 10523 raft_consensus.cc:482] T 9546fd87f6e148e492af92b357f65fc1 P b54d024ec6964d07844331e5a81db31e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:15:29.782898 10523 raft_consensus.cc:504] T 9546fd87f6e148e492af92b357f65fc1 P b54d024ec6964d07844331e5a81db31e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } } peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } }
I20260926 06:15:29.783049 10523 leader_election.cc:288] T 9546fd87f6e148e492af92b357f65fc1 P b54d024ec6964d07844331e5a81db31e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56036654e641486b99a44119fcff490f (127.2.110.193:37843), df439d14be854ee5a671c9817319a4d4 (127.2.110.195:34709)
I20260926 06:15:29.783274 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9546fd87f6e148e492af92b357f65fc1" candidate_uuid: "b54d024ec6964d07844331e5a81db31e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df439d14be854ee5a671c9817319a4d4" is_pre_election: true
I20260926 06:15:29.783322 10159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9546fd87f6e148e492af92b357f65fc1" candidate_uuid: "b54d024ec6964d07844331e5a81db31e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56036654e641486b99a44119fcff490f" is_pre_election: true
I20260926 06:15:29.783399 10159 raft_consensus.cc:2454] T 9546fd87f6e148e492af92b357f65fc1 P 56036654e641486b99a44119fcff490f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b54d024ec6964d07844331e5a81db31e in term 0.
I20260926 06:15:29.783390 10438 raft_consensus.cc:2454] T 9546fd87f6e148e492af92b357f65fc1 P df439d14be854ee5a671c9817319a4d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b54d024ec6964d07844331e5a81db31e in term 0.
I20260926 06:15:29.783547 10231 leader_election.cc:302] T 9546fd87f6e148e492af92b357f65fc1 P b54d024ec6964d07844331e5a81db31e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b54d024ec6964d07844331e5a81db31e, df439d14be854ee5a671c9817319a4d4; no voters:
I20260926 06:15:29.783636 10523 raft_consensus.cc:2781] T 9546fd87f6e148e492af92b357f65fc1 P b54d024ec6964d07844331e5a81db31e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:15:29.783679 10523 raft_consensus.cc:482] T 9546fd87f6e148e492af92b357f65fc1 P b54d024ec6964d07844331e5a81db31e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:15:29.783702 10523 raft_consensus.cc:3037] T 9546fd87f6e148e492af92b357f65fc1 P b54d024ec6964d07844331e5a81db31e [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:29.784538 10523 raft_consensus.cc:504] T 9546fd87f6e148e492af92b357f65fc1 P b54d024ec6964d07844331e5a81db31e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } } peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } }
I20260926 06:15:29.784698 10523 leader_election.cc:288] T 9546fd87f6e148e492af92b357f65fc1 P b54d024ec6964d07844331e5a81db31e [CANDIDATE]: Term 1 election: Requested vote from peers 56036654e641486b99a44119fcff490f (127.2.110.193:37843), df439d14be854ee5a671c9817319a4d4 (127.2.110.195:34709)
I20260926 06:15:29.784883 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9546fd87f6e148e492af92b357f65fc1" candidate_uuid: "b54d024ec6964d07844331e5a81db31e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df439d14be854ee5a671c9817319a4d4"
I20260926 06:15:29.784889 10159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9546fd87f6e148e492af92b357f65fc1" candidate_uuid: "b54d024ec6964d07844331e5a81db31e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56036654e641486b99a44119fcff490f"
I20260926 06:15:29.784976 10159 raft_consensus.cc:3037] T 9546fd87f6e148e492af92b357f65fc1 P 56036654e641486b99a44119fcff490f [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:29.784984 10438 raft_consensus.cc:3037] T 9546fd87f6e148e492af92b357f65fc1 P df439d14be854ee5a671c9817319a4d4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:29.785709 10438 raft_consensus.cc:2454] T 9546fd87f6e148e492af92b357f65fc1 P df439d14be854ee5a671c9817319a4d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b54d024ec6964d07844331e5a81db31e in term 1.
I20260926 06:15:29.785807 10159 raft_consensus.cc:2454] T 9546fd87f6e148e492af92b357f65fc1 P 56036654e641486b99a44119fcff490f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b54d024ec6964d07844331e5a81db31e in term 1.
I20260926 06:15:29.785905 10231 leader_election.cc:302] T 9546fd87f6e148e492af92b357f65fc1 P b54d024ec6964d07844331e5a81db31e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b54d024ec6964d07844331e5a81db31e, df439d14be854ee5a671c9817319a4d4; no voters:
I20260926 06:15:29.786032 10523 raft_consensus.cc:2781] T 9546fd87f6e148e492af92b357f65fc1 P b54d024ec6964d07844331e5a81db31e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:15:29.786175 10523 raft_consensus.cc:686] T 9546fd87f6e148e492af92b357f65fc1 P b54d024ec6964d07844331e5a81db31e [term 1 LEADER]: Becoming Leader. State: Replica: b54d024ec6964d07844331e5a81db31e, State: Running, Role: LEADER
I20260926 06:15:29.786237 10523 consensus_queue.cc:237] T 9546fd87f6e148e492af92b357f65fc1 P b54d024ec6964d07844331e5a81db31e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } } peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } }
I20260926 06:15:29.786962 10000 catalog_manager.cc:5495] T 9546fd87f6e148e492af92b357f65fc1 P b54d024ec6964d07844331e5a81db31e reported cstate change: term changed from 0 to 1, leader changed from <none> to b54d024ec6964d07844331e5a81db31e (127.2.110.194). New cstate: current_term: 1 leader_uuid: "b54d024ec6964d07844331e5a81db31e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b54d024ec6964d07844331e5a81db31e" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 36115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 } health_report { overall_health: UNKNOWN } } }
I20260926 06:15:29.814851 10584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:29.812326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33438 (local address 127.2.110.194:36115)
0926 06:15:29.812395 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:29.812396 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:29.812405 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:15:29.812474 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:29.812475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:29.812484 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:29.812529 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:15:29.812759 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:29.813519 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:29.814266 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:29.814457 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:29.814508 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:29.814613 (+ 105us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:15:29.814638 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:15:29.814670 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:29.814702 (+ 32us) server_negotiation.cc:299] Negotiation successful
0926 06:15:29.814720 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 06:15:29.819906 10511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:29.817688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:35225 (local address 127.2.110.195:34709)
0926 06:15:29.817786 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:29.817788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:29.817798 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:15:29.817847 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:29.817848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:29.817861 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:29.817910 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:15:29.818015 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:29.818551 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:29.819321 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:29.819505 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:29.819568 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:29.819698 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:15:29.819723 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:15:29.819762 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:29.819830 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 06:15:29.819846 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 06:15:29.819924 10600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:29.817593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.195:34709 (local address 127.2.110.194:35225)
0926 06:15:29.817727 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:29.817739 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:29.817793 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:29.817914 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:29.817914 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:29.817915 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 06:15:29.817957 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:29.817958 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:29.818659 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:29.818660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:29.819241 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:29.819243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:29.819579 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:29.819579 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:29.819618 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:29.819631 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 06:15:29.819787 (+ 156us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 06:15:29.819789 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 06:15:29.819824 (+ 35us) client_negotiation.cc:240] Negotiation successful
0926 06:15:29.819863 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":77,"threads_started":1}
I20260926 06:15:29.822131 10438 raft_consensus.cc:1261] T 9546fd87f6e148e492af92b357f65fc1 P df439d14be854ee5a671c9817319a4d4 [term 1 FOLLOWER]: Refusing update from remote peer b54d024ec6964d07844331e5a81db31e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:15:29.822166 10159 raft_consensus.cc:1261] T 9546fd87f6e148e492af92b357f65fc1 P 56036654e641486b99a44119fcff490f [term 1 FOLLOWER]: Refusing update from remote peer b54d024ec6964d07844331e5a81db31e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:15:29.822371 10596 consensus_queue.cc:1035] T 9546fd87f6e148e492af92b357f65fc1 P b54d024ec6964d07844331e5a81db31e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df439d14be854ee5a671c9817319a4d4" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:15:29.822515 10523 consensus_queue.cc:1035] T 9546fd87f6e148e492af92b357f65fc1 P b54d024ec6964d07844331e5a81db31e [LEADER]: Connected to new peer: Peer: permanent_uuid: "56036654e641486b99a44119fcff490f" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 37843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:15:29.830317 10584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:29.827473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:47163 (local address 127.2.110.194:36115)
0926 06:15:29.827550 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:29.827553 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:29.827688 (+ 135us) server_negotiation.cc:407] Connection header received
0926 06:15:29.827777 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:29.827777 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:29.827792 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:29.827847 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:15:29.827952 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:29.828633 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:29.829724 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:29.829979 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:29.830039 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:29.830146 (+ 107us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:15:29.830173 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:15:29.830205 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:29.830235 (+ 30us) server_negotiation.cc:299] Negotiation successful
0926 06:15:29.830261 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 06:15:29.830338 10605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:29.827432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.194:36115 (local address 127.2.110.195:47163)
0926 06:15:29.827598 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:29.827612 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:29.827704 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:29.827854 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:29.827855 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:29.827857 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 06:15:29.827904 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:29.827905 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:29.828711 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:29.828711 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:29.829575 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:29.829577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:29.830049 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:29.830049 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:29.830090 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:29.830094 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 06:15:29.830204 (+ 110us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 06:15:29.830205 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 06:15:29.830234 (+ 29us) client_negotiation.cc:240] Negotiation successful
0926 06:15:29.830279 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":79,"threads_started":1}
I20260926 06:15:29.836351 10001 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:15:29.836474 10001 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:15:29.837114 10001 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 24536bc278a44d40bfcee7eb68808bf6: Sending DeleteTablet for 3 replicas of tablet 9546fd87f6e148e492af92b357f65fc1
I20260926 06:15:29.837442 10279 tablet_service.cc:1514] Processing DeleteTablet for tablet 9546fd87f6e148e492af92b357f65fc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:15:29 UTC) from {username='oryx', principal='oryx/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:33412
I20260926 06:15:29.837435 10139 tablet_service.cc:1514] Processing DeleteTablet for tablet 9546fd87f6e148e492af92b357f65fc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:15:29 UTC) from {username='oryx', principal='oryx/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:34980
I20260926 06:15:29.837486 10419 tablet_service.cc:1514] Processing DeleteTablet for tablet 9546fd87f6e148e492af92b357f65fc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:15:29 UTC) from {username='oryx', principal='oryx/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:41970
I20260926 06:15:29.837540 10535 tablet_replica.cc:330] T 9546fd87f6e148e492af92b357f65fc1 P b54d024ec6964d07844331e5a81db31e: stopping tablet replica
I20260926 06:15:29.837567 10536 tablet_replica.cc:330] T 9546fd87f6e148e492af92b357f65fc1 P 56036654e641486b99a44119fcff490f: stopping tablet replica
I20260926 06:15:29.837589 10535 raft_consensus.cc:2229] T 9546fd87f6e148e492af92b357f65fc1 P b54d024ec6964d07844331e5a81db31e [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:15:29.837630 10536 raft_consensus.cc:2229] T 9546fd87f6e148e492af92b357f65fc1 P 56036654e641486b99a44119fcff490f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:15:29.837646 10535 raft_consensus.cc:2258] T 9546fd87f6e148e492af92b357f65fc1 P b54d024ec6964d07844331e5a81db31e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:15:29.837661 10536 raft_consensus.cc:2258] T 9546fd87f6e148e492af92b357f65fc1 P 56036654e641486b99a44119fcff490f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:15:29.837716 10537 tablet_replica.cc:330] T 9546fd87f6e148e492af92b357f65fc1 P df439d14be854ee5a671c9817319a4d4: stopping tablet replica
I20260926 06:15:29.837777 10537 raft_consensus.cc:2229] T 9546fd87f6e148e492af92b357f65fc1 P df439d14be854ee5a671c9817319a4d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:15:29.837822 10537 raft_consensus.cc:2258] T 9546fd87f6e148e492af92b357f65fc1 P df439d14be854ee5a671c9817319a4d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:15:29.838093 10535 ts_tablet_manager.cc:1905] T 9546fd87f6e148e492af92b357f65fc1 P b54d024ec6964d07844331e5a81db31e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:15:29.838174 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 10074
I20260926 06:15:29.838199 10536 ts_tablet_manager.cc:1905] T 9546fd87f6e148e492af92b357f65fc1 P 56036654e641486b99a44119fcff490f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:15:29.840022 10535 ts_tablet_manager.cc:1918] T 9546fd87f6e148e492af92b357f65fc1 P b54d024ec6964d07844331e5a81db31e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 06:15:29.840080 10535 log.cc:1196] T 9546fd87f6e148e492af92b357f65fc1 P b54d024ec6964d07844331e5a81db31e: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-1/wal/wals/9546fd87f6e148e492af92b357f65fc1
I20260926 06:15:29.840299 10535 ts_tablet_manager.cc:1939] T 9546fd87f6e148e492af92b357f65fc1 P b54d024ec6964d07844331e5a81db31e: Deleting consensus metadata
I20260926 06:15:29.840757 9986 catalog_manager.cc:4841] TS b54d024ec6964d07844331e5a81db31e (127.2.110.194:36115): tablet 9546fd87f6e148e492af92b357f65fc1 (table test-table [id=a74c161978604e30b61457185bbd0b04]) successfully deleted
I20260926 06:15:29.845803 10537 ts_tablet_manager.cc:1905] T 9546fd87f6e148e492af92b357f65fc1 P df439d14be854ee5a671c9817319a4d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:15:29.847431 10537 ts_tablet_manager.cc:1918] T 9546fd87f6e148e492af92b357f65fc1 P df439d14be854ee5a671c9817319a4d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 06:15:29.847579 10537 log.cc:1196] T 9546fd87f6e148e492af92b357f65fc1 P df439d14be854ee5a671c9817319a4d4: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403292369990-2491-0/minicluster-data/ts-2/wal/wals/9546fd87f6e148e492af92b357f65fc1
I20260926 06:15:29.847892 10537 ts_tablet_manager.cc:1939] T 9546fd87f6e148e492af92b357f65fc1 P df439d14be854ee5a671c9817319a4d4: Deleting consensus metadata
W20260926 06:15:29.848026 9986 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.110.193:37843 (error 108)
I20260926 06:15:29.848302 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 10214
I20260926 06:15:29.848349 9987 catalog_manager.cc:4841] TS df439d14be854ee5a671c9817319a4d4 (127.2.110.195:34709): tablet 9546fd87f6e148e492af92b357f65fc1 (table test-table [id=a74c161978604e30b61457185bbd0b04]) successfully deleted
I20260926 06:15:29.849098 10607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:29.848423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.193:37843 (local address 127.0.0.1:35006)
0926 06:15:29.848993 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:29.849048 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.193:37843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":39,"threads_started":1}
W20260926 06:15:29.850020 9986 catalog_manager.cc:4584] TS 56036654e641486b99a44119fcff490f (127.2.110.193:37843): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9546fd87f6e148e492af92b357f65fc1: Network error: Client connection negotiation failed: client connection to 127.2.110.193:37843: connect: Connection refused (error 111)
I20260926 06:15:29.854856 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 10354
I20260926 06:15:29.861553 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 9970
2026-09-26T06:15:29Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3032 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790403292369990-2491-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:15:29 dist-test-slave-hndd krb5kdc[10610](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:15:29 dist-test-slave-hndd krb5kdc[10610](info): set up 2 sockets
Sep 26 06:15:29 dist-test-slave-hndd krb5kdc[10610](info): commencing operation
krb5kdc: starting...
W20260926 06:15:30.949999 2491 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Sep 26 06:15:31 dist-test-slave-hndd krb5kdc[10610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:15:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:15:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
I20260926 06:15:31.076918 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:36603
--webserver_interface=127.2.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:36103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.110.254:36603
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:15:31.163436 10623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:31.163717 10623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:31.163769 10623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:31.164979 10623 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:15:31.165055 10623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:31.165081 10623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:31.165102 10623 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:15:31.165130 10623 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:15:31.166764 10623 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:36103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790403292369990-2491-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.110.254:36603
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:36603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:31.167068 10623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:31.167306 10623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:31.172328 10631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:31.172468 10623 server_base.cc:1034] running on GCE node
W20260926 06:15:31.172358 10628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:31.172362 10629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:31.172863 10623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:31.173189 10623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:31.174361 10623 hybrid_clock.cc:648] HybridClock initialized: now 1790403331174350 us; error 31 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T06:15:31Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1322 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:15:31 dist-test-slave-hndd krb5kdc[10636](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:15:31 dist-test-slave-hndd krb5kdc[10636](info): set up 2 sockets
Sep 26 06:15:31 dist-test-slave-hndd krb5kdc[10636](info): commencing operation
krb5kdc: starting...
W20260926 06:15:32.230079 2491 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Sep 26 06:15:32 dist-test-slave-hndd krb5kdc[10636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:15:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:15:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
I20260926 06:15:32.366127 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:34933
--webserver_interface=127.2.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:34885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.110.254:34933
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:15:32.454504 10649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:32.454762 10649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:32.454800 10649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:32.455993 10649 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:15:32.456067 10649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:32.456100 10649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:32.456127 10649 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:15:32.456159 10649 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:15:32.457731 10649 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:34885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.110.254:34933
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:34933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:32.458011 10649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:32.458281 10649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:32.463459 10657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:32.463459 10655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:32.463668 10649 server_base.cc:1034] running on GCE node
W20260926 06:15:32.463497 10654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:32.464005 10649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:32.464301 10649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:32.465438 10649 hybrid_clock.cc:648] HybridClock initialized: now 1790403332465425 us; error 29 us; skew 500 ppm
Sep 26 06:15:32 dist-test-slave-hndd krb5kdc[10636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403332, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:32.471182 10649 init.cc:377] Logged in from keytab as kudu/127.2.110.254@KRBTEST.COM (short username kudu)
I20260926 06:15:32.471762 10649 webserver.cc:460] Webserver started at http://127.2.110.254:44583/ using document root <none> and password file <none>
I20260926 06:15:32.471956 10649 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:32.472005 10649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:32.472098 10649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:32.473205 10649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/master-0/data/instance:
uuid: "8785268b196749408e9f6a6ab2024f98"
format_stamp: "Formatted at 2026-09-26 06:15:32 on dist-test-slave-hndd"
server_key: "047d09449a3ba14325e997f3f762f2fe"
server_key_iv: "2dd9b3a626e120418186c8daaeae6c34"
server_key_version: "encryptionkey@0"
I20260926 06:15:32.473583 10649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/master-0/wal/instance:
uuid: "8785268b196749408e9f6a6ab2024f98"
format_stamp: "Formatted at 2026-09-26 06:15:32 on dist-test-slave-hndd"
server_key: "047d09449a3ba14325e997f3f762f2fe"
server_key_iv: "2dd9b3a626e120418186c8daaeae6c34"
server_key_version: "encryptionkey@0"
I20260926 06:15:32.474880 10649 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:32.475683 10664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:32.475893 10649 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:32.476001 10649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "8785268b196749408e9f6a6ab2024f98"
format_stamp: "Formatted at 2026-09-26 06:15:32 on dist-test-slave-hndd"
server_key: "047d09449a3ba14325e997f3f762f2fe"
server_key_iv: "2dd9b3a626e120418186c8daaeae6c34"
server_key_version: "encryptionkey@0"
I20260926 06:15:32.476078 10649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:32.482897 10649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:32.484889 10649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:32.485060 10649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:32.489244 10649 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.254:34933
I20260926 06:15:32.489279 10716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.254:34933 every 8 connection(s)
I20260926 06:15:32.489674 10649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:15:32.490502 10717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:32.491540 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 10649
I20260926 06:15:32.491629 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/master-0/wal/instance
I20260926 06:15:32.491856 2491 external_mini_cluster.cc:1428] Setting key 2e57236eb0118b690fc3bdd9dd48d8d4
I20260926 06:15:32.493177 10717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8785268b196749408e9f6a6ab2024f98: Bootstrap starting.
I20260926 06:15:32.493947 10717 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8785268b196749408e9f6a6ab2024f98: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:32.494273 10717 log.cc:824] T 00000000000000000000000000000000 P 8785268b196749408e9f6a6ab2024f98: Log is configured to *not* fsync() on all Append() calls
I20260926 06:15:32.494967 10717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8785268b196749408e9f6a6ab2024f98: No bootstrap required, opened a new log
I20260926 06:15:32.497463 10717 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8785268b196749408e9f6a6ab2024f98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8785268b196749408e9f6a6ab2024f98" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 34933 } }
I20260926 06:15:32.497658 10717 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8785268b196749408e9f6a6ab2024f98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:32.497702 10717 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8785268b196749408e9f6a6ab2024f98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8785268b196749408e9f6a6ab2024f98, State: Initialized, Role: FOLLOWER
I20260926 06:15:32.497854 10717 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8785268b196749408e9f6a6ab2024f98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8785268b196749408e9f6a6ab2024f98" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 34933 } }
I20260926 06:15:32.497947 10717 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8785268b196749408e9f6a6ab2024f98 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:15:32.497987 10717 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8785268b196749408e9f6a6ab2024f98 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:15:32.498041 10717 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8785268b196749408e9f6a6ab2024f98 [term 0 FOLLOWER]: Advancing to term 1
Sep 26 06:15:32 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:15:32.498862 10717 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8785268b196749408e9f6a6ab2024f98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8785268b196749408e9f6a6ab2024f98" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 34933 } }
I20260926 06:15:32.499022 10717 leader_election.cc:302] T 00000000000000000000000000000000 P 8785268b196749408e9f6a6ab2024f98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8785268b196749408e9f6a6ab2024f98; no voters:
I20260926 06:15:32.499226 10717 leader_election.cc:288] T 00000000000000000000000000000000 P 8785268b196749408e9f6a6ab2024f98 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:15:32.499277 10722 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8785268b196749408e9f6a6ab2024f98 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:15:32.499432 10722 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8785268b196749408e9f6a6ab2024f98 [term 1 LEADER]: Becoming Leader. State: Replica: 8785268b196749408e9f6a6ab2024f98, State: Running, Role: LEADER
I20260926 06:15:32.499526 10717 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8785268b196749408e9f6a6ab2024f98 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:15:32.499607 10722 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8785268b196749408e9f6a6ab2024f98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8785268b196749408e9f6a6ab2024f98" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 34933 } }
I20260926 06:15:32.500058 10723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8785268b196749408e9f6a6ab2024f98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8785268b196749408e9f6a6ab2024f98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8785268b196749408e9f6a6ab2024f98" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 34933 } } }
I20260926 06:15:32.500159 10723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8785268b196749408e9f6a6ab2024f98 [sys.catalog]: This master's current role is: LEADER
I20260926 06:15:32.500386 10724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8785268b196749408e9f6a6ab2024f98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8785268b196749408e9f6a6ab2024f98. Latest consensus state: current_term: 1 leader_uuid: "8785268b196749408e9f6a6ab2024f98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8785268b196749408e9f6a6ab2024f98" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 34933 } } }
I20260926 06:15:32.500414 10731 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:15:32.500455 10724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8785268b196749408e9f6a6ab2024f98 [sys.catalog]: This master's current role is: LEADER
I20260926 06:15:32.500957 10731 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:15:32.503338 10720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:32.492382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48860 (local address 127.2.110.254:34933)
0926 06:15:32.492555 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:32.492562 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:32.492587 (+ 25us) server_negotiation.cc:407] Connection header received
0926 06:15:32.493011 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:32.493017 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:32.493062 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:32.493175 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:32.494019 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:32.495146 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:32.496149 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:32.496561 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:32.496641 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:32.499458 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:32.499462 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:32.499462 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:32.499474 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:32.501567 (+ 2093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:32.502039 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:32.502043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:32.502044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:32.502152 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:32.502376 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:32.502377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:32.502377 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:32.502651 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:32.502827 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:32.502975 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 06:15:32.503130 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":91,"threads_started":1}
I20260926 06:15:32.503666 10731 catalog_manager.cc:1347] Generated new cluster ID: e8d0d4bf5dbd4dcf83d1aca35cfdb032
I20260926 06:15:32.503726 10731 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:15:32.517769 10731 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:15:32.518353 10731 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:15:32.527920 10731 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8785268b196749408e9f6a6ab2024f98: Generated new TSK 0
I20260926 06:15:32.528146 10731 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab.
Entry for principal kudu/127.2.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab.
I20260926 06:15:32.557379 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.193:0
--local_ip_for_outbound_sockets=127.2.110.193
--webserver_interface=127.2.110.193
--webserver_port=0
--tserver_master_addrs=127.2.110.254:34933
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab
--principal=kudu/127.2.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:34885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:15:32.643496 10743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:32.643832 10743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:32.643877 10743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:32.645921 10743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:32.645989 10743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:32.646041 10743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.193
I20260926 06:15:32.647765 10743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:34885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab
--principal=kudu/127.2.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.110.193
--webserver_port=0
--tserver_master_addrs=127.2.110.254:34933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.193
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:32.647986 10743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:32.648173 10743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:32.653396 10751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:32.653422 10749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:32.653402 10748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:32.653565 10743 server_base.cc:1034] running on GCE node
I20260926 06:15:32.653800 10743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:32.654066 10743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:32.655261 10743 hybrid_clock.cc:648] HybridClock initialized: now 1790403332655248 us; error 29 us; skew 500 ppm
Sep 26 06:15:32 dist-test-slave-hndd krb5kdc[10636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403332, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:32.657912 10743 init.cc:377] Logged in from keytab as kudu/127.2.110.193@KRBTEST.COM (short username kudu)
I20260926 06:15:32.658377 10743 webserver.cc:460] Webserver started at http://127.2.110.193:35045/ using document root <none> and password file <none>
I20260926 06:15:32.658601 10743 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:32.658706 10743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:32.658844 10743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:32.659935 10743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-0/data/instance:
uuid: "ce117306d1034899b8e5033b52574a2b"
format_stamp: "Formatted at 2026-09-26 06:15:32 on dist-test-slave-hndd"
server_key: "dae88eb918a5e9e7bd33a9afab5a7afa"
server_key_iv: "2a6c8080b180e613b7ba883e751aef85"
server_key_version: "encryptionkey@0"
I20260926 06:15:32.660321 10743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-0/wal/instance:
uuid: "ce117306d1034899b8e5033b52574a2b"
format_stamp: "Formatted at 2026-09-26 06:15:32 on dist-test-slave-hndd"
server_key: "dae88eb918a5e9e7bd33a9afab5a7afa"
server_key_iv: "2a6c8080b180e613b7ba883e751aef85"
server_key_version: "encryptionkey@0"
I20260926 06:15:32.661731 10743 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:15:32.663425 10758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:32.663633 10743 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:15:32.663739 10743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-0/wal
uuid: "ce117306d1034899b8e5033b52574a2b"
format_stamp: "Formatted at 2026-09-26 06:15:32 on dist-test-slave-hndd"
server_key: "dae88eb918a5e9e7bd33a9afab5a7afa"
server_key_iv: "2a6c8080b180e613b7ba883e751aef85"
server_key_version: "encryptionkey@0"
I20260926 06:15:32.663821 10743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:32.679253 10743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:32.681284 10743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:32.681504 10743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:32.681725 10743 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:15:32.682092 10743 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:15:32.682170 10743 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:32.682241 10743 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:15:32.682286 10743 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:32.688301 10743 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.193:34373
I20260926 06:15:32.688341 10871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.193:34373 every 8 connection(s)
I20260926 06:15:32.688948 10743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
I20260926 06:15:32.693075 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 10743
I20260926 06:15:32.693187 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-0/wal/instance
I20260926 06:15:32.693450 2491 external_mini_cluster.cc:1428] Setting key f0c2a493328fc3cd97198385817050d0
Sep 26 06:15:32 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403332, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.193@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:15:32.699180 10720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:32.689881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:52043 (local address 127.2.110.254:34933)
0926 06:15:32.689947 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:32.689949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:32.690172 (+ 223us) server_negotiation.cc:407] Connection header received
0926 06:15:32.690725 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:32.690726 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:32.690736 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:32.690806 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:32.691810 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:32.692322 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:32.693460 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:32.693784 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:32.693881 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:32.696134 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:32.696136 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:32.696136 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:32.696143 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:32.697710 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:32.698143 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:32.698144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:32.698145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:32.698208 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:32.698474 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:32.698475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:32.698475 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:32.698600 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:32.698677 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:32.698995 (+ 318us) server_negotiation.cc:299] Negotiation successful
0926 06:15:32.699085 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 06:15:32.699415 10874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:32.689836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34933 (local address 127.2.110.193:52043)
0926 06:15:32.690033 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:32.690078 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:32.690626 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:32.690883 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:32.690886 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:32.691475 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:32.691675 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:32.691679 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:32.692417 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:32.692418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:32.693382 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:32.693385 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:32.693891 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:32.693892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:32.694031 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:32.694633 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:32.694642 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:32.696032 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:32.697811 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:32.697818 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:32.697820 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:32.698050 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:32.698276 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:32.698277 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:32.698277 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:32.698396 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:32.698668 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:32.698674 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:32.698848 (+ 174us) client_negotiation.cc:769] Sending connection context
0926 06:15:32.698974 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 06:15:32.699142 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":124,"threads_started":1}
I20260926 06:15:32.700153 10872 heartbeater.cc:347] Connected to a master server at 127.2.110.254:34933
I20260926 06:15:32.700263 10872 heartbeater.cc:464] Registering TS with master...
I20260926 06:15:32.700469 10872 heartbeater.cc:511] Master 127.2.110.254:34933 requested a full tablet report, sending...
I20260926 06:15:32.701122 10681 ts_manager.cc:194] Registered new tserver with Master: ce117306d1034899b8e5033b52574a2b (127.2.110.193:34373)
WARNING: no policy specified for kudu/127.2.110.194@KRBTEST.COM; defaulting to no policy
I20260926 06:15:32.701970 10681 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.193@KRBTEST.COM'} at 127.2.110.193:52043
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab.
Entry for principal kudu/127.2.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab.
I20260926 06:15:32.722347 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.194:0
--local_ip_for_outbound_sockets=127.2.110.194
--webserver_interface=127.2.110.194
--webserver_port=0
--tserver_master_addrs=127.2.110.254:34933
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab
--principal=kudu/127.2.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:34885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:15:32.809069 10877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:32.809383 10877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:32.809422 10877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:32.811121 10877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:32.811172 10877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:32.811211 10877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.194
I20260926 06:15:32.813315 10877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:34885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab
--principal=kudu/127.2.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.110.194
--webserver_port=0
--tserver_master_addrs=127.2.110.254:34933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.194
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:32.813618 10877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:32.813880 10877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:32.819409 10883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:32.819396 10885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:32.819365 10882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:32.819798 10877 server_base.cc:1034] running on GCE node
I20260926 06:15:32.820052 10877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:32.820331 10877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:32.821533 10877 hybrid_clock.cc:648] HybridClock initialized: now 1790403332821504 us; error 43 us; skew 500 ppm
Sep 26 06:15:32 dist-test-slave-hndd krb5kdc[10636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403332, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:32.824901 10877 init.cc:377] Logged in from keytab as kudu/127.2.110.194@KRBTEST.COM (short username kudu)
I20260926 06:15:32.825476 10877 webserver.cc:460] Webserver started at http://127.2.110.194:38373/ using document root <none> and password file <none>
I20260926 06:15:32.825682 10877 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:32.825726 10877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:32.825816 10877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:32.826877 10877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-1/data/instance:
uuid: "2fe734a770564f3d856288594bc84c31"
format_stamp: "Formatted at 2026-09-26 06:15:32 on dist-test-slave-hndd"
server_key: "2f562adf25a1047ab75a35eaf244962c"
server_key_iv: "9e5bcc2f5116234d03ee11f3e6cb47b1"
server_key_version: "encryptionkey@0"
I20260926 06:15:32.827207 10877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-1/wal/instance:
uuid: "2fe734a770564f3d856288594bc84c31"
format_stamp: "Formatted at 2026-09-26 06:15:32 on dist-test-slave-hndd"
server_key: "2f562adf25a1047ab75a35eaf244962c"
server_key_iv: "9e5bcc2f5116234d03ee11f3e6cb47b1"
server_key_version: "encryptionkey@0"
I20260926 06:15:32.828488 10877 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:15:32.829187 10892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:32.829389 10877 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:15:32.829476 10877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-1/wal
uuid: "2fe734a770564f3d856288594bc84c31"
format_stamp: "Formatted at 2026-09-26 06:15:32 on dist-test-slave-hndd"
server_key: "2f562adf25a1047ab75a35eaf244962c"
server_key_iv: "9e5bcc2f5116234d03ee11f3e6cb47b1"
server_key_version: "encryptionkey@0"
I20260926 06:15:32.829535 10877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:32.848151 10877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:32.850739 10877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:32.850878 10877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:32.851083 10877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:15:32.851390 10877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:15:32.851423 10877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:32.851445 10877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:15:32.851459 10877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:32.857225 10877 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.194:35273
I20260926 06:15:32.857328 11005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.194:35273 every 8 connection(s)
I20260926 06:15:32.857697 10877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
I20260926 06:15:32.863205 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 10877
I20260926 06:15:32.863338 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-1/wal/instance
I20260926 06:15:32.863550 2491 external_mini_cluster.cc:1428] Setting key 057c00f50f8b2e509d701fc0d86ebc06
Sep 26 06:15:32 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403332, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.194@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:15:32.868295 10720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:32.858481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:57423 (local address 127.2.110.254:34933)
0926 06:15:32.858566 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:32.858568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:32.858807 (+ 239us) server_negotiation.cc:407] Connection header received
0926 06:15:32.859377 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:32.859378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:32.859390 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:32.859450 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:32.860536 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:32.861040 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:32.862070 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:32.862264 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:32.862340 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:32.864945 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:32.864946 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:32.864946 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:32.864953 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:32.866507 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:32.867127 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:32.867128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:32.867128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:32.867181 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:32.867478 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:32.867478 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:32.867479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:32.867602 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:32.867662 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:32.868123 (+ 461us) server_negotiation.cc:299] Negotiation successful
0926 06:15:32.868228 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 06:15:32.868549 11008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:32.858425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34933 (local address 127.2.110.194:57423)
0926 06:15:32.858624 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:32.858687 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:32.859261 (+ 574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:32.859524 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:32.859528 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:32.860206 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:32.860424 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:32.860429 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:32.861127 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:32.861127 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:32.861981 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:32.861984 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:32.862360 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:32.862361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:32.862469 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:32.863036 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:32.863043 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:32.864847 (+ 1804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:32.866609 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:32.866618 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:32.866621 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:32.867040 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:32.867252 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:32.867253 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:32.867253 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:32.867405 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:32.867673 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:32.867679 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:32.867868 (+ 189us) client_negotiation.cc:769] Sending connection context
0926 06:15:32.868045 (+ 177us) client_negotiation.cc:240] Negotiation successful
0926 06:15:32.868252 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":101,"threads_started":1}
I20260926 06:15:32.869074 11006 heartbeater.cc:347] Connected to a master server at 127.2.110.254:34933
I20260926 06:15:32.869197 11006 heartbeater.cc:464] Registering TS with master...
I20260926 06:15:32.869467 11006 heartbeater.cc:511] Master 127.2.110.254:34933 requested a full tablet report, sending...
I20260926 06:15:32.869894 10681 ts_manager.cc:194] Registered new tserver with Master: 2fe734a770564f3d856288594bc84c31 (127.2.110.194:35273)
I20260926 06:15:32.870283 10681 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.194@KRBTEST.COM'} at 127.2.110.194:57423
WARNING: no policy specified for kudu/127.2.110.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab.
Entry for principal kudu/127.2.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab.
I20260926 06:15:32.892927 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.195:0
--local_ip_for_outbound_sockets=127.2.110.195
--webserver_interface=127.2.110.195
--webserver_port=0
--tserver_master_addrs=127.2.110.254:34933
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab
--principal=kudu/127.2.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:34885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:15:32.981997 11011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:32.982237 11011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:32.982264 11011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:32.983501 11011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:32.983559 11011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:32.983589 11011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.195
I20260926 06:15:32.985136 11011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:34885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab
--principal=kudu/127.2.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.110.195
--webserver_port=0
--tserver_master_addrs=127.2.110.254:34933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.195
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:32.985435 11011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:32.985723 11011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:32.991005 11017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:32.991005 11016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:32.991010 11019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:32.991400 11011 server_base.cc:1034] running on GCE node
I20260926 06:15:32.991668 11011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:32.991925 11011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:32.993077 11011 hybrid_clock.cc:648] HybridClock initialized: now 1790403332993065 us; error 30 us; skew 500 ppm
Sep 26 06:15:32 dist-test-slave-hndd krb5kdc[10636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403332, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:32.996094 11011 init.cc:377] Logged in from keytab as kudu/127.2.110.195@KRBTEST.COM (short username kudu)
I20260926 06:15:32.996644 11011 webserver.cc:460] Webserver started at http://127.2.110.195:33095/ using document root <none> and password file <none>
I20260926 06:15:32.996850 11011 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:32.996896 11011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:32.996994 11011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:32.998162 11011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-2/data/instance:
uuid: "2088387ac46a4f4ea67b1e53c77891e7"
format_stamp: "Formatted at 2026-09-26 06:15:32 on dist-test-slave-hndd"
server_key: "4f5de98cf99e3231c6a2b84507a1c165"
server_key_iv: "a7771e93c3bcb353eb67fb2f4d4987b0"
server_key_version: "encryptionkey@0"
I20260926 06:15:32.998487 11011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-2/wal/instance:
uuid: "2088387ac46a4f4ea67b1e53c77891e7"
format_stamp: "Formatted at 2026-09-26 06:15:32 on dist-test-slave-hndd"
server_key: "4f5de98cf99e3231c6a2b84507a1c165"
server_key_iv: "a7771e93c3bcb353eb67fb2f4d4987b0"
server_key_version: "encryptionkey@0"
I20260926 06:15:33.000015 11011 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:15:33.001044 11026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:33.001240 11011 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:33.001343 11011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-2/wal
uuid: "2088387ac46a4f4ea67b1e53c77891e7"
format_stamp: "Formatted at 2026-09-26 06:15:32 on dist-test-slave-hndd"
server_key: "4f5de98cf99e3231c6a2b84507a1c165"
server_key_iv: "a7771e93c3bcb353eb67fb2f4d4987b0"
server_key_version: "encryptionkey@0"
I20260926 06:15:33.001459 11011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:33.016561 11011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:33.018826 11011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:33.019011 11011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:33.019274 11011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:15:33.019613 11011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:15:33.019671 11011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:33.019716 11011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:15:33.019740 11011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:33.025938 11011 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.195:39157
I20260926 06:15:33.026137 11139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.195:39157 every 8 connection(s)
I20260926 06:15:33.026408 11011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
I20260926 06:15:33.028350 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 11011
I20260926 06:15:33.028460 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-2/wal/instance
I20260926 06:15:33.028653 2491 external_mini_cluster.cc:1428] Setting key 6577c3a6d3b4181bec88926f2d8beb4f
Sep 26 06:15:33 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403332, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.195@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:15:33.038319 10720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:33.027299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:34087 (local address 127.2.110.254:34933)
0926 06:15:33.027346 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:33.027348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:33.027575 (+ 227us) server_negotiation.cc:407] Connection header received
0926 06:15:33.028196 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:33.028197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:33.028207 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:33.028268 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:33.029704 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:33.030216 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:33.031260 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:33.031466 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:33.031553 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:33.034125 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:33.034127 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:33.034127 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:33.034141 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:33.036007 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:33.036671 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:33.036673 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:33.036674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:33.036747 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:33.037051 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:33.037052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:33.037052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:33.037206 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:33.037288 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:33.037938 (+ 650us) server_negotiation.cc:299] Negotiation successful
0926 06:15:33.038185 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 06:15:33.038771 11142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:33.027281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34933 (local address 127.2.110.195:34087)
0926 06:15:33.027443 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:33.027479 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:33.028090 (+ 611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:33.028349 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:33.028353 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:33.029289 (+ 936us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:33.029600 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:33.029607 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:33.030329 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:33.030330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:33.031148 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:33.031151 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:33.031561 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:33.031562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:33.031707 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:33.032324 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:33.032334 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:33.033988 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:33.036182 (+ 2194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:33.036198 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:33.036202 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:33.036571 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:33.036829 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:33.036830 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:33.036830 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:33.036974 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:33.037286 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:33.037295 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:33.037738 (+ 443us) client_negotiation.cc:769] Sending connection context
0926 06:15:33.037913 (+ 175us) client_negotiation.cc:240] Negotiation successful
0926 06:15:33.038321 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":95,"threads_started":1}
I20260926 06:15:33.039440 11140 heartbeater.cc:347] Connected to a master server at 127.2.110.254:34933
I20260926 06:15:33.039547 11140 heartbeater.cc:464] Registering TS with master...
I20260926 06:15:33.039770 11140 heartbeater.cc:511] Master 127.2.110.254:34933 requested a full tablet report, sending...
I20260926 06:15:33.040202 10681 ts_manager.cc:194] Registered new tserver with Master: 2088387ac46a4f4ea67b1e53c77891e7 (127.2.110.195:39157)
I20260926 06:15:33.040620 10681 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.195@KRBTEST.COM'} at 127.2.110.195:34087
I20260926 06:15:33.041231 2491 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/oryx_127.2.110.193.keytab.
Entry for principal oryx/127.2.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/oryx_127.2.110.193.keytab.
WARNING: no policy specified for oryx/127.2.110.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.110.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/oryx_127.2.110.194.keytab.
Entry for principal oryx/127.2.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/oryx_127.2.110.194.keytab.
WARNING: no policy specified for oryx/127.2.110.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.110.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/oryx_127.2.110.195.keytab.
Entry for principal oryx/127.2.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/oryx_127.2.110.195.keytab.
I20260926 06:15:33.119877 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 10743
I20260926 06:15:33.124794 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 10877
I20260926 06:15:33.130035 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 11011
I20260926 06:15:33.134850 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 10649
I20260926 06:15:33.140504 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:34933
--webserver_interface=127.2.110.254
--webserver_port=44583
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:34885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.110.254:34933
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:15:33.222558 11149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:33.222795 11149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:33.222819 11149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:33.223968 11149 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:15:33.224007 11149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:33.224021 11149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:33.224035 11149 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:15:33.224046 11149 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:15:33.225601 11149 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:34885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.110.254:34933
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:34933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.110.254
--webserver_port=44583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:33.225818 11149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:33.225994 11149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:33.230862 11155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:33.230870 11157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:33.230876 11154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:33.231113 11149 server_base.cc:1034] running on GCE node
I20260926 06:15:33.231383 11149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:33.231685 11149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:33.232874 11149 hybrid_clock.cc:648] HybridClock initialized: now 1790403333232866 us; error 42 us; skew 500 ppm
Sep 26 06:15:33 dist-test-slave-hndd krb5kdc[10636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403333, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:33.235596 11149 init.cc:377] Logged in from keytab as kudu/127.2.110.254@KRBTEST.COM (short username kudu)
I20260926 06:15:33.236078 11149 webserver.cc:460] Webserver started at http://127.2.110.254:44583/ using document root <none> and password file <none>
I20260926 06:15:33.236323 11149 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:33.236408 11149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:33.237938 11149 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 06:15:33.238646 11164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:33.238862 11149 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:15:33.238967 11149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "8785268b196749408e9f6a6ab2024f98"
format_stamp: "Formatted at 2026-09-26 06:15:32 on dist-test-slave-hndd"
server_key: "047d09449a3ba14325e997f3f762f2fe"
server_key_iv: "2dd9b3a626e120418186c8daaeae6c34"
server_key_version: "encryptionkey@0"
I20260926 06:15:33.239327 11149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:33.247444 11149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:33.249505 11149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:33.249675 11149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:33.253316 11149 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.254:34933
I20260926 06:15:33.253396 11216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.254:34933 every 8 connection(s)
I20260926 06:15:33.253789 11149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:15:33.255261 11217 sys_catalog.cc:263] Verifying existing consensus state
I20260926 06:15:33.255460 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 11149
I20260926 06:15:33.255756 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.193:34373
--local_ip_for_outbound_sockets=127.2.110.193
--tserver_master_addrs=127.2.110.254:34933
--webserver_port=35045
--webserver_interface=127.2.110.193
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab
--principal=kudu/127.2.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:34885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.110.193
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/oryx_127.2.110.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 06:15:33.256094 11217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8785268b196749408e9f6a6ab2024f98: Bootstrap starting.
I20260926 06:15:33.264681 11217 log.cc:824] T 00000000000000000000000000000000 P 8785268b196749408e9f6a6ab2024f98: Log is configured to *not* fsync() on all Append() calls
I20260926 06:15:33.266616 11217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8785268b196749408e9f6a6ab2024f98: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 06:15:33.266938 11217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8785268b196749408e9f6a6ab2024f98: Bootstrap complete.
I20260926 06:15:33.269409 11217 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8785268b196749408e9f6a6ab2024f98 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8785268b196749408e9f6a6ab2024f98" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 34933 } }
I20260926 06:15:33.269663 11217 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8785268b196749408e9f6a6ab2024f98 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8785268b196749408e9f6a6ab2024f98, State: Initialized, Role: FOLLOWER
I20260926 06:15:33.269841 11217 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8785268b196749408e9f6a6ab2024f98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8785268b196749408e9f6a6ab2024f98" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 34933 } }
I20260926 06:15:33.269909 11217 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8785268b196749408e9f6a6ab2024f98 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:15:33.269972 11217 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8785268b196749408e9f6a6ab2024f98 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:15:33.270023 11217 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8785268b196749408e9f6a6ab2024f98 [term 1 FOLLOWER]: Advancing to term 2
I20260926 06:15:33.271046 11217 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8785268b196749408e9f6a6ab2024f98 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8785268b196749408e9f6a6ab2024f98" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 34933 } }
I20260926 06:15:33.271134 11217 leader_election.cc:302] T 00000000000000000000000000000000 P 8785268b196749408e9f6a6ab2024f98 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8785268b196749408e9f6a6ab2024f98; no voters:
I20260926 06:15:33.271355 11217 leader_election.cc:288] T 00000000000000000000000000000000 P 8785268b196749408e9f6a6ab2024f98 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 06:15:33.271394 11221 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8785268b196749408e9f6a6ab2024f98 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 06:15:33.271507 11221 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8785268b196749408e9f6a6ab2024f98 [term 2 LEADER]: Becoming Leader. State: Replica: 8785268b196749408e9f6a6ab2024f98, State: Running, Role: LEADER
I20260926 06:15:33.271595 11221 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8785268b196749408e9f6a6ab2024f98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8785268b196749408e9f6a6ab2024f98" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 34933 } }
I20260926 06:15:33.271623 11217 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8785268b196749408e9f6a6ab2024f98 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:15:33.271853 11221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8785268b196749408e9f6a6ab2024f98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8785268b196749408e9f6a6ab2024f98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8785268b196749408e9f6a6ab2024f98" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 34933 } } }
I20260926 06:15:33.271920 11221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8785268b196749408e9f6a6ab2024f98 [sys.catalog]: This master's current role is: LEADER
I20260926 06:15:33.272034 11221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8785268b196749408e9f6a6ab2024f98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8785268b196749408e9f6a6ab2024f98. Latest consensus state: current_term: 2 leader_uuid: "8785268b196749408e9f6a6ab2024f98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8785268b196749408e9f6a6ab2024f98" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 34933 } } }
I20260926 06:15:33.272097 11221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8785268b196749408e9f6a6ab2024f98 [sys.catalog]: This master's current role is: LEADER
I20260926 06:15:33.272557 11225 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:15:33.273062 11225 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:15:33.273216 11225 catalog_manager.cc:1259] Loaded cluster ID: e8d0d4bf5dbd4dcf83d1aca35cfdb032
I20260926 06:15:33.273510 11225 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:15:33.274693 11225 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:15:33.274860 11225 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8785268b196749408e9f6a6ab2024f98: Loaded TSK: 0
I20260926 06:15:33.275063 11225 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 06:15:33.342180 11219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:33.342442 11219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:33.342484 11219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:33.343937 11219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:33.344015 11219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:33.344065 11219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.193
I20260926 06:15:33.345781 11219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:34885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/oryx_127.2.110.193.keytab
--principal=oryx/127.2.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.193:34373
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.110.193
--webserver_port=35045
--tserver_master_addrs=127.2.110.254:34933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.193
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:33.346077 11219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:33.346630 11219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:33.352118 11246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:33.352149 11243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:33.352294 11219 server_base.cc:1034] running on GCE node
W20260926 06:15:33.352406 11244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:33.352656 11219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:33.352941 11219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:33.354126 11219 hybrid_clock.cc:648] HybridClock initialized: now 1790403333354107 us; error 34 us; skew 500 ppm
Sep 26 06:15:33 dist-test-slave-hndd krb5kdc[10636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403333, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:33.357252 11219 init.cc:377] Logged in from keytab as oryx/127.2.110.193@KRBTEST.COM (short username oryx)
I20260926 06:15:33.357776 11219 webserver.cc:460] Webserver started at http://127.2.110.193:35045/ using document root <none> and password file <none>
I20260926 06:15:33.358031 11219 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:33.358134 11219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:33.359534 11219 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:33.360224 11253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:33.360423 11219 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:33.360533 11219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-0/wal
uuid: "ce117306d1034899b8e5033b52574a2b"
format_stamp: "Formatted at 2026-09-26 06:15:32 on dist-test-slave-hndd"
server_key: "dae88eb918a5e9e7bd33a9afab5a7afa"
server_key_iv: "2a6c8080b180e613b7ba883e751aef85"
server_key_version: "encryptionkey@0"
I20260926 06:15:33.360904 11219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:33.373930 11219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:33.376305 11219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:33.376480 11219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:33.376770 11219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:15:33.377144 11219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:15:33.377199 11219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:33.377239 11219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:15:33.377261 11219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:33.383735 11219 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.193:34373
I20260926 06:15:33.383780 11366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.193:34373 every 8 connection(s)
I20260926 06:15:33.384173 11219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
I20260926 06:15:33.391187 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 11219
I20260926 06:15:33.391553 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.194:35273
--local_ip_for_outbound_sockets=127.2.110.194
--tserver_master_addrs=127.2.110.254:34933
--webserver_port=38373
--webserver_interface=127.2.110.194
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab
--principal=kudu/127.2.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:34885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.110.194
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/oryx_127.2.110.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 06:15:33 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.193@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:15:33 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.193@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:15:33.396711 11369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:33.385036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34933 (local address 127.2.110.193:36333)
0926 06:15:33.385228 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:33.385279 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:33.385811 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:33.386594 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:33.386600 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:33.387401 (+ 801us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:33.387716 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:33.387725 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:33.389436 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:33.389438 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:33.390279 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:33.390281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:33.391137 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:33.391138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:33.391304 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:33.391969 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:33.391980 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:33.396455 (+ 4475us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.110.254:34933: Server oryx/127.2.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":85,"threads_started":1}
W20260926 06:15:33.396836 11370 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:33.385166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:36333 (local address 127.2.110.254:34933)
0926 06:15:33.385403 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:33.385412 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:33.385453 (+ 41us) server_negotiation.cc:407] Connection header received
0926 06:15:33.386024 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:33.386031 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:33.386091 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:33.386239 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:33.387862 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:33.389231 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:33.390374 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:33.391029 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:33.391139 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:33.396653 (+ 5514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.110.193:36333: BlockingRecv error: recv got EOF from 127.2.110.193:36333 (error 108)
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":103,"threads_started":1}
W20260926 06:15:33.397392 11367 heartbeater.cc:644] Failed to heartbeat to 127.2.110.254:34933 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.110.254:34933: Client connection negotiation failed: client connection to 127.2.110.254:34933: Server oryx/127.2.110.254@KRBTEST.COM not found in Kerberos database
Sep 26 06:15:33 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.193@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:15:33 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.193@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:15:33.404030 11369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:33.397676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34933 (local address 127.2.110.193:56735)
0926 06:15:33.397738 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:33.397752 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:33.397812 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:33.397996 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:33.397997 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:33.398184 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:33.398262 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:33.398265 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:33.399199 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:33.399203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:33.400361 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:33.400367 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:33.400872 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:33.400873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:33.400964 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:33.401525 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:33.401531 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:33.403917 (+ 2386us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.110.254:34933: Server oryx/127.2.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 06:15:33.404100 11370 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:33.397801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:56735 (local address 127.2.110.254:34933)
0926 06:15:33.397845 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:33.397848 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:33.397858 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:15:33.397887 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:33.397888 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:33.397904 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:33.397978 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:33.398356 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:33.399040 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:33.400468 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:33.400780 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:33.400849 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:33.404038 (+ 3189us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.110.193:56735: BlockingRecv error: recv got EOF from 127.2.110.193:56735 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 26 06:15:33 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.193@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:15:33 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.193@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:15:33.409904 11369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:33.404437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34933 (local address 127.2.110.193:39901)
0926 06:15:33.404587 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:33.404599 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:33.404665 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:33.404810 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:33.404811 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:33.404994 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:33.405042 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:33.405044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:33.405857 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:33.405858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:33.406800 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:33.406804 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:33.407302 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:33.407303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:33.407363 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:33.407837 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:33.407842 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:33.409811 (+ 1969us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.110.254:34933: Server oryx/127.2.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129}
W20260926 06:15:33.409953 11370 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:33.404536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:39901 (local address 127.2.110.254:34933)
0926 06:15:33.404601 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:33.404603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:33.404662 (+ 59us) server_negotiation.cc:407] Connection header received
0926 06:15:33.404733 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:33.404733 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:33.404744 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:33.404803 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:33.405126 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:33.405774 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:33.406902 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:33.407159 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:33.407240 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:33.409890 (+ 2650us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.110.193:39901: BlockingRecv error: recv got EOF from 127.2.110.193:39901 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260926 06:15:33.410064 11367 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 06:15:33.485738 11371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:33.485999 11371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:33.486040 11371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:33.487267 11371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:33.487341 11371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:33.487388 11371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.194
I20260926 06:15:33.488948 11371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:34885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/oryx_127.2.110.194.keytab
--principal=oryx/127.2.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.194:35273
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.110.194
--webserver_port=38373
--tserver_master_addrs=127.2.110.254:34933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.194
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:33.489221 11371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:33.489480 11371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:33.494534 11379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:33.494606 11377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:33.494544 11376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:33.494935 11371 server_base.cc:1034] running on GCE node
I20260926 06:15:33.495147 11371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:33.495419 11371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:33.496608 11371 hybrid_clock.cc:648] HybridClock initialized: now 1790403333496606 us; error 47 us; skew 500 ppm
Sep 26 06:15:33 dist-test-slave-hndd krb5kdc[10636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403333, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:33.499253 11371 init.cc:377] Logged in from keytab as oryx/127.2.110.194@KRBTEST.COM (short username oryx)
I20260926 06:15:33.499715 11371 webserver.cc:460] Webserver started at http://127.2.110.194:38373/ using document root <none> and password file <none>
I20260926 06:15:33.499945 11371 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:33.500000 11371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:33.501590 11371 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:33.502244 11386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:33.502456 11371 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:33.502547 11371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-1/wal
uuid: "2fe734a770564f3d856288594bc84c31"
format_stamp: "Formatted at 2026-09-26 06:15:32 on dist-test-slave-hndd"
server_key: "2f562adf25a1047ab75a35eaf244962c"
server_key_iv: "9e5bcc2f5116234d03ee11f3e6cb47b1"
server_key_version: "encryptionkey@0"
I20260926 06:15:33.502915 11371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:33.522240 11371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:33.524700 11371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:33.524828 11371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:33.525120 11371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:15:33.525496 11371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:15:33.525535 11371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:33.525557 11371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:15:33.525570 11371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:33.531409 11371 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.194:35273
I20260926 06:15:33.531445 11499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.194:35273 every 8 connection(s)
I20260926 06:15:33.531857 11371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
Sep 26 06:15:33 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.194@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:15:33 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.194@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:15:33.539592 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 11371
I20260926 06:15:33.539932 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.195:39157
--local_ip_for_outbound_sockets=127.2.110.195
--tserver_master_addrs=127.2.110.254:34933
--webserver_port=33095
--webserver_interface=127.2.110.195
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab
--principal=kudu/127.2.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:34885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.110.195
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/oryx_127.2.110.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:15:33.540139 11370 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:33.532647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:52177 (local address 127.2.110.254:34933)
0926 06:15:33.532717 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:33.532720 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:33.532936 (+ 216us) server_negotiation.cc:407] Connection header received
0926 06:15:33.533541 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:33.533543 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:33.533560 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:33.533614 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:33.534619 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:33.535118 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:33.536160 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:33.536377 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:33.536466 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:33.540059 (+ 3593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.110.194:52177: BlockingRecv error: recv got EOF from 127.2.110.194:52177 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 06:15:33.540202 11502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:33.532616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34933 (local address 127.2.110.194:52177)
0926 06:15:33.532812 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:33.532848 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:33.533427 (+ 579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:33.533691 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:33.533695 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:33.534305 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:33.534498 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:33.534502 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:33.535216 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:33.535217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:33.536060 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:33.536062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:33.536478 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:33.536479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:33.536624 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:33.537139 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:33.537147 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:33.539917 (+ 2770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.110.254:34933: Server oryx/127.2.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":88,"threads_started":1}
W20260926 06:15:33.540489 11500 heartbeater.cc:644] Failed to heartbeat to 127.2.110.254:34933 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.110.254:34933: Client connection negotiation failed: client connection to 127.2.110.254:34933: Server oryx/127.2.110.254@KRBTEST.COM not found in Kerberos database
Sep 26 06:15:33 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.194@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:15:33 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.194@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:15:33.545960 11502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:33.540873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34933 (local address 127.2.110.194:56443)
0926 06:15:33.540921 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:33.540931 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:33.540974 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:33.541117 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:33.541118 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:33.541244 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:33.541298 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:33.541299 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:33.542105 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:33.542106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:33.542987 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:33.542990 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:33.543415 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:33.543416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:33.543495 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:33.544023 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:33.544028 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:33.545866 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.110.254:34933: Server oryx/127.2.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 06:15:33.546015 11370 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:33.540929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:56443 (local address 127.2.110.254:34933)
0926 06:15:33.540981 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:33.540983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:33.540994 (+ 11us) server_negotiation.cc:407] Connection header received
0926 06:15:33.541023 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:33.541024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:33.541035 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:33.541106 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:33.541451 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:33.541989 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:33.543090 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:33.543337 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:33.543393 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:33.545952 (+ 2559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.110.194:56443: BlockingRecv error: recv got EOF from 127.2.110.194:56443 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 26 06:15:33 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.194@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:15:33 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.194@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:15:33.551005 11502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:33.546257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34933 (local address 127.2.110.194:60001)
0926 06:15:33.546301 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:33.546325 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:33.546364 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:33.546531 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:33.546538 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:33.546645 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:33.546676 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:33.546678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:33.547658 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:33.547658 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:33.548152 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:33.548153 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:33.548645 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:33.548645 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:33.548706 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:33.549156 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:33.549162 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:33.550915 (+ 1753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.110.254:34933: Server oryx/127.2.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 06:15:33.551060 11370 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:33.546325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:60001 (local address 127.2.110.254:34933)
0926 06:15:33.546390 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:33.546392 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:33.546403 (+ 11us) server_negotiation.cc:407] Connection header received
0926 06:15:33.546429 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:33.546430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:33.546442 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:33.546526 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:33.546747 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:33.547547 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:33.548356 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:33.548558 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:33.548635 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:33.550992 (+ 2357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.110.194:60001: BlockingRecv error: recv got EOF from 127.2.110.194:60001 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260926 06:15:33.551221 11500 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 06:15:33.627539 11503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:33.627802 11503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:33.627836 11503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:33.629103 11503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:33.629164 11503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:33.629197 11503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.195
I20260926 06:15:33.630805 11503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:34885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/krb5kdc/oryx_127.2.110.195.keytab
--principal=oryx/127.2.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.195:39157
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.110.195
--webserver_port=33095
--tserver_master_addrs=127.2.110.254:34933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.195
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:33.631074 11503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:33.631351 11503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:33.636560 11509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:33.636560 11508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:33.636582 11511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:33.636945 11503 server_base.cc:1034] running on GCE node
I20260926 06:15:33.637171 11503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:33.637480 11503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:33.638679 11503 hybrid_clock.cc:648] HybridClock initialized: now 1790403333638660 us; error 29 us; skew 500 ppm
Sep 26 06:15:33 dist-test-slave-hndd krb5kdc[10636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403333, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:33.641250 11503 init.cc:377] Logged in from keytab as oryx/127.2.110.195@KRBTEST.COM (short username oryx)
I20260926 06:15:33.641732 11503 webserver.cc:460] Webserver started at http://127.2.110.195:33095/ using document root <none> and password file <none>
I20260926 06:15:33.641973 11503 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:33.642055 11503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:33.643513 11503 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:33.644137 11518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:33.644305 11503 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:33.644413 11503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-2/wal
uuid: "2088387ac46a4f4ea67b1e53c77891e7"
format_stamp: "Formatted at 2026-09-26 06:15:32 on dist-test-slave-hndd"
server_key: "4f5de98cf99e3231c6a2b84507a1c165"
server_key_iv: "a7771e93c3bcb353eb67fb2f4d4987b0"
server_key_version: "encryptionkey@0"
I20260926 06:15:33.644804 11503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:33.654760 11503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:33.656982 11503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:33.657181 11503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:33.657521 11503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:15:33.657871 11503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:15:33.657927 11503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:33.657968 11503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:15:33.657996 11503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:33.664153 11503 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.195:39157
I20260926 06:15:33.664193 11631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.195:39157 every 8 connection(s)
I20260926 06:15:33.664592 11503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
I20260926 06:15:33.665936 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 11503
Sep 26 06:15:33 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.195@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:15:33 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.195@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 06:15:33.675349 11370 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:33.665431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:51149 (local address 127.2.110.254:34933)
0926 06:15:33.665481 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:33.665484 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:33.665724 (+ 240us) server_negotiation.cc:407] Connection header received
0926 06:15:33.666809 (+ 1085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:33.666810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:33.666822 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:33.666886 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:33.668304 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:33.669096 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:33.670388 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:33.670685 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:33.670758 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:33.675285 (+ 4527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.110.195:51149: BlockingRecv error: recv got EOF from 127.2.110.195:51149 (error 108)
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":22}
I20260926 06:15:33.675603 11634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:33.665408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34933 (local address 127.2.110.195:51149)
0926 06:15:33.665577 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:33.665623 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:33.666432 (+ 809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:33.666999 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:33.667003 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:33.667856 (+ 853us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:33.668154 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:33.668161 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:33.669222 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:33.669223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:33.670298 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:33.670302 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:33.670773 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:33.670774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:33.670880 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:33.671645 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:33.671659 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:33.675216 (+ 3557us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.110.254:34933: Server oryx/127.2.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":96,"threads_started":1}
W20260926 06:15:33.675933 11632 heartbeater.cc:644] Failed to heartbeat to 127.2.110.254:34933 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.110.254:34933: Client connection negotiation failed: client connection to 127.2.110.254:34933: Server oryx/127.2.110.254@KRBTEST.COM not found in Kerberos database
I20260926 06:15:33.676678 11636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:33.666673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48870 (local address 127.2.110.254:34933)
0926 06:15:33.666818 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:33.666819 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:33.667753 (+ 934us) server_negotiation.cc:407] Connection header received
0926 06:15:33.667780 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:33.667781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:33.667794 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:33.667850 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:33.668655 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:33.669491 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:33.670420 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:33.670705 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:33.670758 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:33.672492 (+ 1734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:33.672497 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:33.672497 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:33.672513 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:33.674962 (+ 2449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:33.675446 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:33.675451 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:33.675451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:33.675568 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:33.675815 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:33.675817 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:33.675817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:33.676129 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:33.676281 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:33.676448 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 06:15:33.676572 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":94,"threads_started":1}
Sep 26 06:15:33 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.195@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:15:33 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.195@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:15:33.680986 11634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:33.676270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34933 (local address 127.2.110.195:45151)
0926 06:15:33.676317 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:33.676329 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:33.676371 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:33.676504 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:33.676504 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:33.676665 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:33.676734 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:33.676737 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:33.677662 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:33.677663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:33.678216 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:33.678218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:33.678589 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:33.678590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:33.678641 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:33.679057 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:33.679061 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:33.680933 (+ 1872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.110.254:34933: Server oryx/127.2.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 06:15:33.681114 11370 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:33.676314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:45151 (local address 127.2.110.254:34933)
0926 06:15:33.676387 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:33.676390 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:33.676398 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:15:33.676421 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:33.676421 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:33.676433 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:33.676498 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:33.676828 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:33.677512 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:33.678290 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:33.678486 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:33.678546 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:33.681036 (+ 2490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.110.195:45151: BlockingRecv error: recv got EOF from 127.2.110.195:45151 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 26 06:15:33 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.195@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:15:33 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.195@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:15:33.685467 11634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:33.681394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34933 (local address 127.2.110.195:34509)
0926 06:15:33.681430 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:33.681441 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:33.681486 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:33.681645 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:33.681646 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:33.681772 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:33.681803 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:33.681805 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:33.682402 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:33.682402 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:33.682888 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:33.682889 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:33.683259 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:33.683260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:33.683306 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:33.683717 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:33.683721 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:33.685348 (+ 1627us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.110.254:34933: Server oryx/127.2.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":17}
W20260926 06:15:33.685504 11370 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:33.681427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:34509 (local address 127.2.110.254:34933)
0926 06:15:33.681487 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:33.681489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:33.681537 (+ 48us) server_negotiation.cc:407] Connection header received
0926 06:15:33.681557 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:33.681557 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:33.681566 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:33.681607 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:33.681872 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:33.682325 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:33.682997 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:33.683175 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:33.683231 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:33.685443 (+ 2212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.110.195:34509: BlockingRecv error: recv got EOF from 127.2.110.195:34509 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260926 06:15:33.685649 11632 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 06:15:34 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.193@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:15:34 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.193@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:15:34.416453 11637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:34.410710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34933 (local address 127.2.110.193:59489)
0926 06:15:34.410909 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:34.410925 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:34.411005 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:34.411274 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:34.411276 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:34.411482 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:34.411537 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:34.411539 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:34.412240 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:34.412241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:34.412794 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:34.412795 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:34.413201 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:34.413202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:34.413278 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:34.413834 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:34.413839 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:34.416381 (+ 2542us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.110.254:34933: Server oryx/127.2.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":128,"threads_started":1}
W20260926 06:15:34.416765 11638 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:34.410837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:59489 (local address 127.2.110.254:34933)
0926 06:15:34.410990 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:34.410992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:34.411005 (+ 13us) server_negotiation.cc:407] Connection header received
0926 06:15:34.411117 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:34.411118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:34.411132 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:34.411238 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:34.411631 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:34.412141 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:34.412889 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:34.413093 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:34.413203 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:34.416677 (+ 3474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.110.193:59489: BlockingRecv error: recv got EOF from 127.2.110.193:59489 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":83,"threads_started":1}
Sep 26 06:15:34 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.194@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:15:34 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.194@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:15:34.558001 11639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:34.551978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34933 (local address 127.2.110.194:50851)
0926 06:15:34.552211 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:34.552235 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:34.552350 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:34.552613 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:34.552614 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:34.552797 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:34.552855 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:34.552857 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:34.553831 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:34.553833 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:34.554658 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:34.554661 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:34.555059 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:34.555060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:34.555123 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:34.555705 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:34.555710 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:34.557925 (+ 2215us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.110.254:34933: Server oryx/127.2.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":109,"threads_started":1}
W20260926 06:15:34.558262 11638 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:34.552155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:50851 (local address 127.2.110.254:34933)
0926 06:15:34.552257 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:34.552259 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:34.552410 (+ 151us) server_negotiation.cc:407] Connection header received
0926 06:15:34.552456 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:34.552457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:34.552474 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:34.552551 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:34.552980 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:34.553694 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:34.554753 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:34.554965 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:34.555047 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:34.558167 (+ 3120us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.110.194:50851: BlockingRecv error: recv got EOF from 127.2.110.194:50851 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 26 06:15:34 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.195@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:15:34 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.195@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:15:34.691110 11640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:34.686068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34933 (local address 127.2.110.195:39689)
0926 06:15:34.686196 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:34.686207 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:34.686280 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:34.686468 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:34.686469 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:34.686672 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:34.686717 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:34.686719 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:34.687587 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:34.687588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:34.688131 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:34.688133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:34.688522 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:34.688523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:34.688593 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:34.689034 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:34.689038 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:34.691028 (+ 1990us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.110.254:34933: Server oryx/127.2.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":80,"threads_started":1}
W20260926 06:15:34.691295 11638 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:34.686199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:39689 (local address 127.2.110.254:34933)
0926 06:15:34.686270 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:34.686272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:34.686292 (+ 20us) server_negotiation.cc:407] Connection header received
0926 06:15:34.686363 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:34.686364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:34.686378 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:34.686448 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:34.686804 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:34.687510 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:34.688233 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:34.688436 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:34.688522 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:34.691176 (+ 2654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.110.195:39689: BlockingRecv error: recv got EOF from 127.2.110.195:39689 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 26 06:15:35 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.193@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:15:35 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.193@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:15:35.423099 11641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:35.417157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34933 (local address 127.2.110.193:37173)
0926 06:15:35.417316 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:35.417356 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:35.417446 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:35.417647 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:35.417648 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:35.417822 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:35.417870 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:35.417872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:35.418676 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:35.418677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:35.419403 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:35.419406 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:35.419842 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:35.419843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:35.419917 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:35.420581 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:35.420588 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:35.423038 (+ 2450us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.110.254:34933: Server oryx/127.2.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":88,"threads_started":1}
W20260926 06:15:35.423416 11642 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:35.417256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:37173 (local address 127.2.110.254:34933)
0926 06:15:35.417419 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:35.417423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:35.417441 (+ 18us) server_negotiation.cc:407] Connection header received
0926 06:15:35.417530 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:35.417531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:35.417541 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:35.417631 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:35.417949 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:35.418460 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:35.419519 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:35.419755 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:35.419846 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:35.423328 (+ 3482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.110.193:37173: BlockingRecv error: recv got EOF from 127.2.110.193:37173 (error 108)
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":128,"threads_started":1}
Sep 26 06:15:35 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.194@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:15:35 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.194@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:15:35.563401 11643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:35.558632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34933 (local address 127.2.110.194:60615)
0926 06:15:35.558832 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:35.558844 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:35.558939 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:35.559101 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:35.559102 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:35.559292 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:35.559339 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:35.559342 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:35.560020 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:35.560021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:35.560584 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:35.560585 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:35.560952 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:35.560953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:35.561009 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:35.561489 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:35.561494 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:35.563340 (+ 1846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.110.254:34933: Server oryx/127.2.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":119,"threads_started":1}
W20260926 06:15:35.563623 11642 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:35.558742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:60615 (local address 127.2.110.254:34933)
0926 06:15:35.558816 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:35.558819 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:35.558919 (+ 100us) server_negotiation.cc:407] Connection header received
0926 06:15:35.559006 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:35.559007 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:35.559019 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:35.559095 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:35.559420 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:35.559922 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:35.560676 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:35.560863 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:35.560951 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:35.563558 (+ 2607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.110.194:60615: BlockingRecv error: recv got EOF from 127.2.110.194:60615 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 26 06:15:35 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.195@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:15:35 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.195@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:15:35.696309 11644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:35.691721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34933 (local address 127.2.110.195:45523)
0926 06:15:35.691963 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:35.691975 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:35.692045 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:35.692178 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:35.692179 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:35.692352 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:35.692400 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:35.692402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:35.693022 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:35.693022 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:35.693628 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:35.693630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:35.694013 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:35.694014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:35.694067 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:35.694570 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:35.694575 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:35.696248 (+ 1673us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.110.254:34933: Server oryx/127.2.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":182,"threads_started":1}
W20260926 06:15:35.696470 11642 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:35.691854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:45523 (local address 127.2.110.254:34933)
0926 06:15:35.691937 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:35.691939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:35.692047 (+ 108us) server_negotiation.cc:407] Connection header received
0926 06:15:35.692091 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:35.692091 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:35.692103 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:35.692160 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:35.692463 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:35.692949 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:35.693708 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:35.693902 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:35.693962 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:35.696387 (+ 2425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.110.195:45523: BlockingRecv error: recv got EOF from 127.2.110.195:45523 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 26 06:15:36 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.193@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:15:36 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.193@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:15:36.429332 11645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:36.423762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34933 (local address 127.2.110.193:42261)
0926 06:15:36.423900 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:36.423913 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:36.423976 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:36.424213 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:36.424215 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:36.424395 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:36.424449 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:36.424450 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:36.425286 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:36.425289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:36.426021 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:36.426025 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:36.426671 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:36.426672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:36.426733 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:36.427231 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:36.427235 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:36.429234 (+ 1999us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.110.254:34933: Server oryx/127.2.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":89,"threads_started":1}
W20260926 06:15:36.429517 11646 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:36.423901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:42261 (local address 127.2.110.254:34933)
0926 06:15:36.424034 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:36.424036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:36.424046 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:15:36.424079 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:36.424080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:36.424090 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:36.424166 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:36.424600 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:36.425136 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:36.426173 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:36.426564 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:36.426647 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:36.429421 (+ 2774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.110.193:42261: BlockingRecv error: recv got EOF from 127.2.110.193:42261 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":83,"threads_started":1}
Sep 26 06:15:36 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.194@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:15:36 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.194@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:15:36.569149 11647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:36.563970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34933 (local address 127.2.110.194:60571)
0926 06:15:36.564153 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:36.564165 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:36.564235 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:36.564396 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:36.564397 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:36.564624 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:36.564683 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:36.564685 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:36.565466 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:36.565466 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:36.566005 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:36.566007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:36.566361 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:36.566362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:36.566402 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:36.566863 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:36.566868 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:36.569067 (+ 2199us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.110.254:34933: Server oryx/127.2.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":100,"threads_started":1}
W20260926 06:15:36.569298 11646 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:36.564105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:60571 (local address 127.2.110.254:34933)
0926 06:15:36.564156 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:36.564158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:36.564237 (+ 79us) server_negotiation.cc:407] Connection header received
0926 06:15:36.564294 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:36.564295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:36.564309 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:36.564371 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:36.564762 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:36.565329 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:36.566084 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:36.566291 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:36.566349 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:36.569184 (+ 2835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.110.194:60571: BlockingRecv error: recv got EOF from 127.2.110.194:60571 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 26 06:15:36 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.195@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:15:36 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.195@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:15:36.702721 11648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:36.696907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34933 (local address 127.2.110.195:44403)
0926 06:15:36.697036 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:36.697057 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:36.697120 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:36.697279 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:36.697280 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:36.697539 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:36.697601 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:36.697614 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:36.698452 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:36.698453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:36.698978 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:36.698979 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:36.699694 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:36.699695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:36.699744 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:36.700242 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:36.700246 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:36.702639 (+ 2393us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.110.254:34933: Server oryx/127.2.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":82,"threads_started":1}
W20260926 06:15:36.702811 11646 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:36.697068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:44403 (local address 127.2.110.254:34933)
0926 06:15:36.697137 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:36.697139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:36.697149 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:15:36.697178 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:36.697179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:36.697190 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:36.697255 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:36.697722 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:36.698352 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:36.699141 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:36.699560 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:36.699671 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:36.702756 (+ 3085us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.110.195:44403: BlockingRecv error: recv got EOF from 127.2.110.195:44403 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 26 06:15:37 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.193@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:15:37 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.193@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:15:37.434918 11649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:37.429901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34933 (local address 127.2.110.193:49191)
0926 06:15:37.430019 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:37.430038 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:37.430093 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:37.430320 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:37.430321 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:37.430483 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:37.430535 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:37.430536 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:37.431257 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:37.431258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:37.431801 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:37.431802 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:37.432191 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:37.432192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:37.432233 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:37.432788 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:37.432792 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:37.434857 (+ 2065us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.110.254:34933: Server oryx/127.2.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":66,"threads_started":1}
W20260926 06:15:37.435060 11650 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:37.429968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:49191 (local address 127.2.110.254:34933)
0926 06:15:37.430138 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:37.430140 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:37.430149 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:15:37.430168 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:37.430169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:37.430178 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:37.430245 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:37.430641 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:37.431140 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:37.431896 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:37.432115 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:37.432185 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:37.434966 (+ 2781us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.110.193:49191: BlockingRecv error: recv got EOF from 127.2.110.193:49191 (error 108)
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":97,"threads_started":1}
Sep 26 06:15:37 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.194@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:15:37 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.194@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:15:37.579254 11651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:37.569821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34933 (local address 127.2.110.194:40897)
0926 06:15:37.570047 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:37.570060 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:37.570164 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:37.570336 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:37.570337 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:37.570546 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:37.570597 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:37.570599 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:37.571321 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:37.571322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:37.571919 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:37.571920 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:37.572349 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:37.572350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:37.572407 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:37.576801 (+ 4394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:37.576813 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:37.579153 (+ 2340us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.110.254:34933: Server oryx/127.2.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":111,"threads_started":1}
W20260926 06:15:37.579376 11650 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:37.569925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:40897 (local address 127.2.110.254:34933)
0926 06:15:37.570035 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:37.570038 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:37.570145 (+ 107us) server_negotiation.cc:407] Connection header received
0926 06:15:37.570238 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:37.570240 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:37.570259 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:37.570329 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:37.570711 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:37.571219 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:37.572008 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:37.572242 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:37.572351 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:37.579289 (+ 6938us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.110.194:40897: BlockingRecv error: recv got EOF from 127.2.110.194:40897 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 26 06:15:37 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.195@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:15:37 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.195@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:15:37.709494 11652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:37.703636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34933 (local address 127.2.110.195:36855)
0926 06:15:37.703863 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:37.703878 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:37.703967 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:37.704165 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:37.704167 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:37.704353 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:37.704408 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:37.704410 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:37.705555 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:37.705557 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:37.706183 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:37.706186 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:37.706672 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:37.706673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:37.706767 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:37.707355 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:37.707360 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:37.709426 (+ 2066us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.110.254:34933: Server oryx/127.2.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":95,"threads_started":1}
W20260926 06:15:37.709632 11650 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:37.703782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:36855 (local address 127.2.110.254:34933)
0926 06:15:37.703902 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:37.703905 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:37.703947 (+ 42us) server_negotiation.cc:407] Connection header received
0926 06:15:37.704046 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:37.704047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:37.704063 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:37.704132 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:37.704534 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:37.705415 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:37.706326 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:37.706562 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:37.706648 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:37.709542 (+ 2894us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.110.195:36855: BlockingRecv error: recv got EOF from 127.2.110.195:36855 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 26 06:15:38 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.193@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:15:38 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.193@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:15:38.440755 11653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:38.435556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34933 (local address 127.2.110.193:41103)
0926 06:15:38.435727 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:38.435751 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:38.435803 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:38.435958 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:38.435959 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:38.436180 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:38.436252 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:38.436255 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:38.437141 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:38.437142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:38.437737 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:38.437739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:38.438129 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:38.438130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:38.438191 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:38.438729 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:38.438734 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:38.440657 (+ 1923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.110.254:34933: Server oryx/127.2.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":113,"threads_started":1}
W20260926 06:15:38.440809 11654 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:38.435672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:41103 (local address 127.2.110.254:34933)
0926 06:15:38.435797 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:38.435799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:38.435812 (+ 13us) server_negotiation.cc:407] Connection header received
0926 06:15:38.435854 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:38.435854 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:38.435865 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:38.435941 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:38.436348 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:38.437053 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:38.437825 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:38.438035 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:38.438122 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:38.440737 (+ 2615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.110.193:41103: BlockingRecv error: recv got EOF from 127.2.110.193:41103 (error 108)
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":75,"threads_started":1}
Sep 26 06:15:38 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.194@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:15:38 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.194@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:15:38.585327 11655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:38.580000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34933 (local address 127.2.110.194:42571)
0926 06:15:38.580196 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:38.580243 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:38.580323 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:38.580523 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:38.580524 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:38.580711 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:38.580757 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:38.580759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:38.581486 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:38.581487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:38.582022 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:38.582024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:38.582484 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:38.582486 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:38.582580 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:38.583180 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:38.583185 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:38.585264 (+ 2079us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.110.254:34933: Server oryx/127.2.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":99,"threads_started":1}
W20260926 06:15:38.585532 11654 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:38.580187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:42571 (local address 127.2.110.254:34933)
0926 06:15:38.580281 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:38.580283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:38.580300 (+ 17us) server_negotiation.cc:407] Connection header received
0926 06:15:38.580417 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:38.580418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:38.580432 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:38.580512 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:38.580842 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:38.581388 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:38.582159 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:38.582373 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:38.582444 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:38.585467 (+ 3023us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.110.194:42571: BlockingRecv error: recv got EOF from 127.2.110.194:42571 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 26 06:15:38 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.195@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:15:38 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.195@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:15:38.714833 11656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:38.710104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34933 (local address 127.2.110.195:36619)
0926 06:15:38.710272 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:38.710284 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:38.710355 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:38.710520 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:38.710521 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:38.710695 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:38.710754 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:38.710756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:38.711459 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:38.711460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:38.711999 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:38.712001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:38.712328 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:38.712329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:38.712368 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:38.712846 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:38.712850 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:38.714772 (+ 1922us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.110.254:34933: Server oryx/127.2.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":86,"threads_started":1}
W20260926 06:15:38.715018 11654 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:38.710186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:36619 (local address 127.2.110.254:34933)
0926 06:15:38.710285 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:38.710287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:38.710355 (+ 68us) server_negotiation.cc:407] Connection header received
0926 06:15:38.710405 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:38.710406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:38.710417 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:38.710472 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:38.710855 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:38.711355 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:38.712080 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:38.712260 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:38.712315 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:38.714909 (+ 2594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.110.195:36619: BlockingRecv error: recv got EOF from 127.2.110.195:36619 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 26 06:15:39 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.193@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:15:39 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.193@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:15:39.446655 11657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:39.441417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34933 (local address 127.2.110.193:44059)
0926 06:15:39.441572 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:39.441584 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:39.441646 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:39.441798 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:39.441798 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:39.441961 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:39.442012 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:39.442014 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:39.442785 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:39.442786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:39.443397 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:39.443399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:39.443792 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:39.443793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:39.443845 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:39.444377 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:39.444382 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:39.446545 (+ 2163us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.110.254:34933: Server oryx/127.2.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":98,"threads_started":1}
W20260926 06:15:39.446702 11658 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:39.441516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:44059 (local address 127.2.110.254:34933)
0926 06:15:39.441640 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:39.441642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:39.441652 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:15:39.441708 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:39.441709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:39.441719 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:39.441780 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:39.442143 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:39.442704 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:39.443491 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:39.443699 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:39.443756 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:39.446632 (+ 2876us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.110.193:44059: BlockingRecv error: recv got EOF from 127.2.110.193:44059 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":53,"threads_started":1}
Sep 26 06:15:39 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.194@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:15:39 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.194@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:15:39.591316 11659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:39.586069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34933 (local address 127.2.110.194:58353)
0926 06:15:39.586388 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:39.586404 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:39.586512 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:39.586673 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:39.586674 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:39.586879 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:39.586925 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:39.586927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:39.587617 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:39.587618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:39.588137 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:39.588138 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:39.588525 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:39.588526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:39.588584 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:39.589160 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:39.589164 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:39.591215 (+ 2051us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.110.254:34933: Server oryx/127.2.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":187,"threads_started":1}
W20260926 06:15:39.591389 11658 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:39.586252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:58353 (local address 127.2.110.254:34933)
0926 06:15:39.586374 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:39.586377 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:39.586489 (+ 112us) server_negotiation.cc:407] Connection header received
0926 06:15:39.586575 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:39.586576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:39.586590 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:39.586680 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:39.587026 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:39.587528 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:39.588247 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:39.588434 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:39.588520 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:39.591316 (+ 2796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.110.194:58353: BlockingRecv error: recv got EOF from 127.2.110.194:58353 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 26 06:15:39 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.195@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:15:39 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.195@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:15:39.720752 11660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:39.715519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34933 (local address 127.2.110.195:58681)
0926 06:15:39.715757 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:39.715770 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:39.715829 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:39.716070 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:39.716071 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:39.716258 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:39.716332 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:39.716334 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:39.717284 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:39.717285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:39.717991 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:39.717994 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:39.718375 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:39.718376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:39.718428 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:39.718956 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:39.718961 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:39.720690 (+ 1729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.110.254:34933: Server oryx/127.2.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":122,"threads_started":1}
W20260926 06:15:39.720932 11658 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:39.715676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:58681 (local address 127.2.110.254:34933)
0926 06:15:39.715883 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:39.715885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:39.715898 (+ 13us) server_negotiation.cc:407] Connection header received
0926 06:15:39.715929 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:39.715930 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:39.715946 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:39.716017 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:39.716560 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:39.717197 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:39.718095 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:39.718295 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:39.718361 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:39.720857 (+ 2496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.110.195:58681: BlockingRecv error: recv got EOF from 127.2.110.195:58681 (error 108)
Metrics: {"server-negotiator.queue_time_us":164}
Sep 26 06:15:40 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.193@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:15:40 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.193@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:15:40.453207 11661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:40.447515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34933 (local address 127.2.110.193:60191)
0926 06:15:40.447730 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:40.447743 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:40.447834 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:40.448052 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:40.448053 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:40.448238 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:40.448299 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:40.448301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:40.449144 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:40.449145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:40.449759 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:40.449761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:40.450133 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:40.450136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:40.450319 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:40.450960 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:40.450965 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:40.453105 (+ 2140us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.110.254:34933: Server oryx/127.2.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":116,"threads_started":1}
W20260926 06:15:40.453266 11662 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:40.447625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:60191 (local address 127.2.110.254:34933)
0926 06:15:40.447756 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:40.447758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:40.447832 (+ 74us) server_negotiation.cc:407] Connection header received
0926 06:15:40.447900 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:40.447901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:40.447918 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:40.448000 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:40.448500 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:40.449039 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:40.449831 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:40.450017 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:40.450076 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:40.453192 (+ 3116us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.110.193:60191: BlockingRecv error: recv got EOF from 127.2.110.193:60191 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
Sep 26 06:15:40 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.194@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:15:40 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.194@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:15:40.596817 11663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:40.591970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34933 (local address 127.2.110.194:60831)
0926 06:15:40.592127 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:40.592139 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:40.592211 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:40.592359 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:40.592360 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:40.592534 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:40.592584 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:40.592586 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:40.593263 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:40.593264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:40.593900 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:40.593902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:40.594248 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:40.594249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:40.594288 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:40.594805 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:40.594810 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:40.596720 (+ 1910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.110.254:34933: Server oryx/127.2.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":104,"threads_started":1}
W20260926 06:15:40.596930 11662 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:40.592070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:60831 (local address 127.2.110.254:34933)
0926 06:15:40.592145 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:40.592147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:40.592218 (+ 71us) server_negotiation.cc:407] Connection header received
0926 06:15:40.592278 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:40.592279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:40.592290 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:40.592335 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:40.592679 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:40.593177 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:40.593989 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:40.594178 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:40.594236 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:40.596853 (+ 2617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.110.194:60831: BlockingRecv error: recv got EOF from 127.2.110.194:60831 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 26 06:15:40 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.195@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:15:40 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.195@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:15:40.726140 11664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:40.721267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34933 (local address 127.2.110.195:42957)
0926 06:15:40.721441 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:40.721475 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:40.721549 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:40.721704 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:40.721705 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:40.721886 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:40.721939 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:40.721940 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:40.722643 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:40.722644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:40.723181 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:40.723183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:40.723562 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:40.723563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:40.723626 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:40.724173 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:40.724181 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:40.726036 (+ 1855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.110.254:34933: Server oryx/127.2.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":122,"threads_started":1}
W20260926 06:15:40.726199 11662 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:40.721425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:42957 (local address 127.2.110.254:34933)
0926 06:15:40.721502 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:40.721504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:40.721531 (+ 27us) server_negotiation.cc:407] Connection header received
0926 06:15:40.721631 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:40.721632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:40.721645 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:40.721699 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:40.722036 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:40.722548 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:40.723272 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:40.723471 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:40.723561 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:40.726117 (+ 2556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.110.195:42957: BlockingRecv error: recv got EOF from 127.2.110.195:42957 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 26 06:15:41 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.193@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:15:41 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.193@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:15:41.459157 11665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:41.453801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34933 (local address 127.2.110.193:58345)
0926 06:15:41.454044 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:41.454072 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:41.454209 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:41.454419 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:41.454421 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:41.454639 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:41.454715 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:41.454717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:41.455476 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:41.455477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:41.456239 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:41.456242 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:41.456645 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:41.456646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:41.456693 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:41.457252 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:41.457258 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:41.459081 (+ 1823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.110.254:34933: Server oryx/127.2.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":59,"threads_started":1}
W20260926 06:15:41.459339 11666 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:41.453981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:58345 (local address 127.2.110.254:34933)
0926 06:15:41.454116 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:41.454118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:41.454195 (+ 77us) server_negotiation.cc:407] Connection header received
0926 06:15:41.454281 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:41.454282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:41.454292 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:41.454365 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:41.454815 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:41.455367 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:41.456333 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:41.456551 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:41.456626 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:41.459252 (+ 2626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.110.193:58345: BlockingRecv error: recv got EOF from 127.2.110.193:58345 (error 108)
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":69,"threads_started":1}
Sep 26 06:15:41 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.194@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:15:41 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.194@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:15:41.603066 11667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:41.597401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34933 (local address 127.2.110.194:37617)
0926 06:15:41.597569 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:41.597588 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:41.597649 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:41.597775 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:41.597775 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:41.597935 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:41.597983 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:41.597986 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:41.599048 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:41.599049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:41.599907 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:41.599910 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:41.600423 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:41.600424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:41.600477 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:41.600929 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:41.600934 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:41.602970 (+ 2036us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.110.254:34933: Server oryx/127.2.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":113,"threads_started":1}
W20260926 06:15:41.603168 11666 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:41.597478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:37617 (local address 127.2.110.254:34933)
0926 06:15:41.597529 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:41.597532 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:41.597637 (+ 105us) server_negotiation.cc:407] Connection header received
0926 06:15:41.597713 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:41.597713 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:41.597725 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:41.597767 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:41.598137 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:41.598890 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:41.600036 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:41.600346 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:41.600416 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:41.603097 (+ 2681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.110.194:37617: BlockingRecv error: recv got EOF from 127.2.110.194:37617 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 26 06:15:41 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.195@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:15:41 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.195@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:15:41.731417 11668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:41.726732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34933 (local address 127.2.110.195:32867)
0926 06:15:41.726884 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:41.726896 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:41.726958 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:41.727098 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:41.727099 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:41.727256 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:41.727304 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:41.727305 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:41.728046 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:41.728047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:41.728597 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:41.728599 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:41.728957 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:41.728957 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:41.728995 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:41.729572 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:41.729578 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:41.731343 (+ 1765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.110.254:34933: Server oryx/127.2.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":101,"threads_started":1}
W20260926 06:15:41.731586 11666 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:41.726815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:32867 (local address 127.2.110.254:34933)
0926 06:15:41.726918 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:41.726920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:41.726951 (+ 31us) server_negotiation.cc:407] Connection header received
0926 06:15:41.727006 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:41.727007 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:41.727021 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:41.727085 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:41.727395 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:41.727950 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:41.728702 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:41.728885 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:41.728957 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:41.731506 (+ 2549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.110.195:32867: BlockingRecv error: recv got EOF from 127.2.110.195:32867 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 26 06:15:42 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.193@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:15:42 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.193@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:15:42.465468 11669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:42.459842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34933 (local address 127.2.110.193:47069)
0926 06:15:42.459993 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:42.460007 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:42.460065 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:42.460230 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:42.460231 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:42.460415 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:42.460478 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:42.460480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:42.461301 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:42.461301 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:42.462128 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:42.462132 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:42.462497 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:42.462498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:42.462549 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:42.463142 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:42.463148 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:42.465329 (+ 2181us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.110.254:34933: Server oryx/127.2.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":88,"threads_started":1}
W20260926 06:15:42.465615 11670 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:42.459894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:47069 (local address 127.2.110.254:34933)
0926 06:15:42.460033 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:42.460036 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:42.460056 (+ 20us) server_negotiation.cc:407] Connection header received
0926 06:15:42.460118 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:42.460118 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:42.460128 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:42.460215 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:42.460589 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:42.461193 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:42.462202 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:42.462403 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:42.462461 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:42.465515 (+ 3054us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.110.193:47069: BlockingRecv error: recv got EOF from 127.2.110.193:47069 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":79,"threads_started":1}
Sep 26 06:15:42 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.194@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:15:42 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.194@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:15:42.613567 11671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:42.603785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34933 (local address 127.2.110.194:42951)
0926 06:15:42.604040 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:42.604059 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:42.604125 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:42.604273 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:42.604274 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:42.604440 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:42.604489 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:42.604491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:42.605537 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:42.605539 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:42.606513 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:42.606517 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:42.606903 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:42.606904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:42.606952 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:42.611448 (+ 4496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:42.611454 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:42.613494 (+ 2040us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.110.254:34933: Server oryx/127.2.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":201,"threads_started":1}
W20260926 06:15:42.613757 11670 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:42.603943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:42951 (local address 127.2.110.254:34933)
0926 06:15:42.604025 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:42.604027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:42.604119 (+ 92us) server_negotiation.cc:407] Connection header received
0926 06:15:42.604192 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:42.604193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:42.604205 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:42.604251 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:42.604585 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:42.605398 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:42.606618 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:42.606822 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:42.606885 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:42.613647 (+ 6762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.110.194:42951: BlockingRecv error: recv got EOF from 127.2.110.194:42951 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 06:15:42 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.195@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:15:42 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.195@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:15:42.737859 11672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:42.732003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34933 (local address 127.2.110.195:33129)
0926 06:15:42.732169 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:42.732181 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:42.732313 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:42.732520 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:42.732521 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:42.732766 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:42.732844 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:42.732847 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:42.733904 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:42.733906 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:42.734440 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:42.734441 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:42.734768 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:42.734769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:42.734809 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:42.735279 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:42.735284 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:42.737740 (+ 2456us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.110.254:34933: Server oryx/127.2.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":61,"threads_started":1}
W20260926 06:15:42.737960 11670 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:42.732174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:33129 (local address 127.2.110.254:34933)
0926 06:15:42.732249 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:42.732251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:42.732289 (+ 38us) server_negotiation.cc:407] Connection header received
0926 06:15:42.732410 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:42.732411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:42.732426 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:42.732506 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:42.732976 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:42.733797 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:42.734522 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:42.734703 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:42.734756 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:42.737881 (+ 3125us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.110.195:33129: BlockingRecv error: recv got EOF from 127.2.110.195:33129 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 26 06:15:43 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.193@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:15:43 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.193@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:15:43.471804 11673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:43.466098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34933 (local address 127.2.110.193:44225)
0926 06:15:43.466393 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:43.466406 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:43.466467 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:43.466609 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:43.466610 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:43.466776 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:43.466827 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:43.466828 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:43.467508 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:43.467509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:43.468059 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:43.468061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:43.468637 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:43.468638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:43.468689 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:43.469252 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:43.469259 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:43.471649 (+ 2390us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.110.254:34933: Server oryx/127.2.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":208,"threads_started":1}
W20260926 06:15:43.471909 11674 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:43.466324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:44225 (local address 127.2.110.254:34933)
0926 06:15:43.466466 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:43.466469 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:43.466479 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:15:43.466509 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:43.466509 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:43.466519 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:43.466589 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:43.466895 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:43.467413 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:43.468251 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:43.468513 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:43.468615 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:43.471786 (+ 3171us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.110.193:44225: BlockingRecv error: recv got EOF from 127.2.110.193:44225 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":80,"threads_started":1}
Sep 26 06:15:43 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.194@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:15:43 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.194@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:15:43.620211 11675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:43.614229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34933 (local address 127.2.110.194:46881)
0926 06:15:43.614424 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:43.614437 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:43.614548 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:43.614728 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:43.614729 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:43.614920 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:43.614969 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:43.614971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:43.615932 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:43.615934 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:43.616818 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:43.616822 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:43.617442 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:43.617443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:43.617501 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:43.618051 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:43.618056 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:43.620126 (+ 2070us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.110.254:34933: Server oryx/127.2.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":102,"threads_started":1}
W20260926 06:15:43.620472 11674 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:43.614360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:46881 (local address 127.2.110.254:34933)
0926 06:15:43.614428 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:43.614431 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:43.614516 (+ 85us) server_negotiation.cc:407] Connection header received
0926 06:15:43.614634 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:43.614635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:43.614653 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:43.614722 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:43.615103 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:43.615840 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:43.616990 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:43.617290 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:43.617377 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:43.620415 (+ 3038us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.110.194:46881: BlockingRecv error: recv got EOF from 127.2.110.194:46881 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 26 06:15:43 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.195@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:15:43 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.195@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:15:43.743409 11676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:43.738538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34933 (local address 127.2.110.195:44695)
0926 06:15:43.738717 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:43.738730 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:43.738822 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:43.738979 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:43.738980 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:43.739190 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:43.739238 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:43.739239 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:43.739913 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:43.739914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:43.740434 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:43.740436 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:43.740793 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:43.740794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:43.740848 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:43.741459 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:43.741464 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:43.743332 (+ 1868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.110.254:34933: Server oryx/127.2.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":100,"threads_started":1}
W20260926 06:15:43.743539 11674 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:43.738759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:44695 (local address 127.2.110.254:34933)
0926 06:15:43.738830 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:43.738832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:43.738863 (+ 31us) server_negotiation.cc:407] Connection header received
0926 06:15:43.738886 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:43.738887 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:43.738906 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:43.738971 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:43.739321 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:43.739815 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:43.740519 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:43.740715 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:43.740796 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:43.743428 (+ 2632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.110.195:44695: BlockingRecv error: recv got EOF from 127.2.110.195:44695 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 26 06:15:44 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.193@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:15:44 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.193@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:15:44.477164 11677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:44.472477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34933 (local address 127.2.110.193:60845)
0926 06:15:44.472637 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:44.472647 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:44.472720 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:44.472884 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:44.472885 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:44.473058 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:44.473107 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:44.473109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:44.473844 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:44.473845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:44.474507 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:44.474509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:44.474845 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:44.474846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:44.474884 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:44.475315 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:44.475319 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:44.477093 (+ 1774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.110.254:34933: Server oryx/127.2.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":106,"threads_started":1}
W20260926 06:15:44.477557 11678 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:44.472557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:60845 (local address 127.2.110.254:34933)
0926 06:15:44.472663 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:44.472665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:44.472708 (+ 43us) server_negotiation.cc:407] Connection header received
0926 06:15:44.472794 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:44.472795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:44.472812 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:44.472864 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:44.473182 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:44.473768 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:44.474590 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:44.474778 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:44.474834 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:44.477464 (+ 2630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.110.193:60845: BlockingRecv error: recv got EOF from 127.2.110.193:60845 (error 108)
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":61,"threads_started":1}
Sep 26 06:15:44 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.194@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:15:44 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.194@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:15:44.626132 11679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:44.621144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34933 (local address 127.2.110.194:39325)
0926 06:15:44.621312 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:44.621324 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:44.621401 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:44.621593 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:44.621594 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:44.621779 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:44.621830 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:44.621831 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:44.622504 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:44.622505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:44.623070 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:44.623072 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:44.623479 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:44.623480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:44.623536 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:44.624099 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:44.624103 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:44.626053 (+ 1950us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.110.254:34933: Server oryx/127.2.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":107,"threads_started":1}
W20260926 06:15:44.626327 11678 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:44.621247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:39325 (local address 127.2.110.254:34933)
0926 06:15:44.621374 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:44.621376 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:44.621446 (+ 70us) server_negotiation.cc:407] Connection header received
0926 06:15:44.621474 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:44.621475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:44.621486 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:44.621531 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:44.621905 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:44.622408 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:44.623163 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:44.623369 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:44.623447 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:44.626225 (+ 2778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.110.194:39325: BlockingRecv error: recv got EOF from 127.2.110.194:39325 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 26 06:15:44 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.195@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:15:44 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.195@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:15:44.749683 11680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:44.744111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34933 (local address 127.2.110.195:47185)
0926 06:15:44.744303 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:44.744326 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:44.744394 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:44.744551 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:44.744551 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:44.744723 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:44.744767 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:44.744769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:44.745510 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:44.745511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:44.746050 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:44.746051 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:44.746433 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:44.746434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:44.746482 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:44.747080 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:44.747085 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:44.749569 (+ 2484us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.110.254:34933: Server oryx/127.2.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":94,"threads_started":1}
W20260926 06:15:44.749724 11678 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:44.744247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:47185 (local address 127.2.110.254:34933)
0926 06:15:44.744338 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:44.744341 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:44.744379 (+ 38us) server_negotiation.cc:407] Connection header received
0926 06:15:44.744456 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:44.744457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:44.744469 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:44.744528 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:44.744844 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:44.745422 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:44.746155 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:44.746359 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:44.746417 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:44.749657 (+ 3240us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.110.195:47185: BlockingRecv error: recv got EOF from 127.2.110.195:47185 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 06:15:45 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.193@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:15:45 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.193@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:15:45.482908 11681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:45.477767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34933 (local address 127.2.110.193:34261)
0926 06:15:45.477926 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:45.477942 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:45.478001 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:45.478165 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:45.478166 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:45.478331 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:45.478393 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:45.478395 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:45.479085 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:45.479086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:45.479638 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:45.479639 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:45.480008 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:45.480009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:45.480062 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:45.480603 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:45.480610 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:45.482842 (+ 2232us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.110.254:34933: Server oryx/127.2.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":108,"threads_started":1}
W20260926 06:15:45.483057 11682 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:45.477896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:34261 (local address 127.2.110.254:34933)
0926 06:15:45.478031 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:45.478033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:45.478042 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:15:45.478072 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:45.478073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:45.478083 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:45.478149 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:45.478493 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:45.478988 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:45.479723 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:45.479936 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:45.479992 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:45.482957 (+ 2965us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.110.193:34261: BlockingRecv error: recv got EOF from 127.2.110.193:34261 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":77,"threads_started":1}
Sep 26 06:15:45 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.194@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:15:45 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.194@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:15:45.631654 11683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:45.626705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34933 (local address 127.2.110.194:57545)
0926 06:15:45.626945 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:45.626958 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:45.627066 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:45.627237 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:45.627238 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:45.627435 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:45.627494 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:45.627499 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:45.628238 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:45.628239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:45.628764 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:45.628766 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:45.629151 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:45.629152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:45.629209 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:45.629775 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:45.629780 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:45.631544 (+ 1764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.110.254:34933: Server oryx/127.2.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":104,"threads_started":1}
W20260926 06:15:45.631701 11682 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:45.626826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:57545 (local address 127.2.110.254:34933)
0926 06:15:45.626942 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:45.626945 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:45.627045 (+ 100us) server_negotiation.cc:407] Connection header received
0926 06:15:45.627131 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:45.627132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:45.627144 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:45.627218 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:45.627619 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:45.628136 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:45.628856 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:45.629060 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:45.629150 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:45.631636 (+ 2486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.110.194:57545: BlockingRecv error: recv got EOF from 127.2.110.194:57545 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 26 06:15:45 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.195@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:15:45 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.195@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:15:45.756121 11684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:45.750269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34933 (local address 127.2.110.195:35293)
0926 06:15:45.750570 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:45.750591 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:45.750777 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:45.751010 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:45.751012 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:45.751191 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:45.751239 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:45.751241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:45.752258 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:45.752260 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:45.752841 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:45.752844 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:45.753275 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:45.753276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:45.753386 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:45.753955 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:45.753960 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:45.756060 (+ 2100us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.110.254:34933: Server oryx/127.2.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":125,"threads_started":1}
W20260926 06:15:45.756227 11682 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:45.750479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:35293 (local address 127.2.110.254:34933)
0926 06:15:45.750576 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:45.750578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:45.750742 (+ 164us) server_negotiation.cc:407] Connection header received
0926 06:15:45.750856 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:45.750857 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:45.750869 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:45.750952 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:45.751397 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:45.752108 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:45.752961 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:45.753156 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:45.753225 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:45.756157 (+ 2932us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.110.195:35293: BlockingRecv error: recv got EOF from 127.2.110.195:35293 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 06:15:46 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.193@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:15:46 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.193@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:15:46.489181 11685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:46.483581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34933 (local address 127.2.110.193:39799)
0926 06:15:46.483782 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:46.483797 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:46.483884 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:46.484061 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:46.484062 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:46.484231 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:46.484279 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:46.484281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:46.485210 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:46.485212 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:46.485953 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:46.485956 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:46.486448 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:46.486449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:46.486501 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:46.487030 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:46.487035 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:46.489099 (+ 2064us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.110.254:34933: Server oryx/127.2.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":80,"threads_started":1}
W20260926 06:15:46.489387 11686 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:46.483745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:39799 (local address 127.2.110.254:34933)
0926 06:15:46.483885 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:46.483887 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:46.483917 (+ 30us) server_negotiation.cc:407] Connection header received
0926 06:15:46.483937 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:46.483938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:46.483954 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:46.484006 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:46.484449 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:46.485045 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:46.486099 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:46.486349 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:46.486416 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:46.489252 (+ 2836us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.110.193:39799: BlockingRecv error: recv got EOF from 127.2.110.193:39799 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":75,"threads_started":1}
Sep 26 06:15:46 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.194@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:15:46 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.194@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:15:46.637651 11687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:46.632244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34933 (local address 127.2.110.194:58861)
0926 06:15:46.632413 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:46.632426 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:46.632497 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:46.632698 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:46.632699 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:46.632900 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:46.632947 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:46.632949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:46.633690 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:46.633691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:46.634221 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:46.634223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:46.634590 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:46.634591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:46.634645 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:46.635174 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:46.635182 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:46.637557 (+ 2375us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.110.254:34933: Server oryx/127.2.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":63,"threads_started":1}
W20260926 06:15:46.637742 11686 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:46.632496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:58861 (local address 127.2.110.254:34933)
0926 06:15:46.632565 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:46.632567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:46.632574 (+ 7us) server_negotiation.cc:407] Connection header received
0926 06:15:46.632610 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:46.632611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:46.632620 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:46.632690 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:46.633033 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:46.633607 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:46.634320 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:46.634513 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:46.634592 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:46.637680 (+ 3088us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.110.194:58861: BlockingRecv error: recv got EOF from 127.2.110.194:58861 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 26 06:15:46 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.195@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:15:46 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.195@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:15:46.762084 11688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:46.756770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34933 (local address 127.2.110.195:35609)
0926 06:15:46.756966 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:46.756981 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:46.757085 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:46.757285 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:46.757286 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:46.757496 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:46.757550 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:46.757552 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:46.758262 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:46.758263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:46.758814 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:46.758816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:46.759197 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:46.759198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:46.759264 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:46.759755 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:46.759760 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:46.762006 (+ 2246us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.110.254:34933: Server oryx/127.2.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":102,"threads_started":1}
W20260926 06:15:46.762140 11686 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:46.756871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:35609 (local address 127.2.110.254:34933)
0926 06:15:46.756970 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:46.756973 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:46.757066 (+ 93us) server_negotiation.cc:407] Connection header received
0926 06:15:46.757178 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:46.757179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:46.757192 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:46.757260 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:46.757652 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:46.758169 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:46.758905 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:46.759106 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:46.759195 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:46.762091 (+ 2896us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.110.195:35609: BlockingRecv error: recv got EOF from 127.2.110.195:35609 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 26 06:15:47 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.193@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:15:47 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.193@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:15:47.494846 11689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:47.489734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34933 (local address 127.2.110.193:39923)
0926 06:15:47.489925 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:47.489941 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:47.490012 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:47.490161 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:47.490162 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:47.490405 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:47.490454 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:47.490456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:47.491181 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:47.491182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:47.491706 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:47.491708 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:47.492069 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:47.492070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:47.492110 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:47.492600 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:47.492605 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:47.494762 (+ 2157us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.110.254:34933: Server oryx/127.2.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":89,"threads_started":1}
W20260926 06:15:47.495008 11690 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:47.489863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:39923 (local address 127.2.110.254:34933)
0926 06:15:47.489997 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:47.489999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:47.490010 (+ 11us) server_negotiation.cc:407] Connection header received
0926 06:15:47.490067 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:47.490067 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:47.490077 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:47.490195 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:47.490542 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:47.491073 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:47.491798 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:47.491997 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:47.492062 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:47.494917 (+ 2855us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.110.193:39923: BlockingRecv error: recv got EOF from 127.2.110.193:39923 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":73,"threads_started":1}
Sep 26 06:15:47 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.194@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:15:47 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.194@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:15:47.650538 11691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:47.638535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34933 (local address 127.2.110.194:32941)
0926 06:15:47.638732 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:47.638764 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:47.638846 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:47.639008 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:47.639009 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:47.639204 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:47.639249 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:47.639251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:47.639942 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:47.639943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:47.640486 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:47.640488 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:47.640864 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:47.640865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:47.640931 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:47.647779 (+ 6848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:47.647785 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:47.650430 (+ 2645us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.110.254:34933: Server oryx/127.2.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":90,"threads_started":1}
W20260926 06:15:47.650609 11690 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:47.638627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:32941 (local address 127.2.110.254:34933)
0926 06:15:47.638671 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:47.638673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:47.638847 (+ 174us) server_negotiation.cc:407] Connection header received
0926 06:15:47.638910 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:47.638911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:47.638923 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:47.639001 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:47.639334 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:47.639837 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:47.640580 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:47.640777 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:47.640853 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:47.650543 (+ 9690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.110.194:32941: BlockingRecv error: recv got EOF from 127.2.110.194:32941 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Sep 26 06:15:47 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.195@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:15:47 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.195@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:15:47.767820 11692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:47.762944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34933 (local address 127.2.110.195:56547)
0926 06:15:47.763105 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:47.763118 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:47.763187 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:47.763375 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:47.763376 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:47.763547 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:47.763594 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:47.763595 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:47.764378 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:47.764379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:47.764917 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:47.764919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:47.765254 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:47.765254 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:47.765294 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:47.765774 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:47.765778 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:47.767716 (+ 1938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.110.254:34933: Server oryx/127.2.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":105,"threads_started":1}
W20260926 06:15:47.767867 11690 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:47.763023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:56547 (local address 127.2.110.254:34933)
0926 06:15:47.763093 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:47.763096 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:47.763221 (+ 125us) server_negotiation.cc:407] Connection header received
0926 06:15:47.763269 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:47.763269 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:47.763284 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:47.763347 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:47.763712 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:47.764286 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:47.765002 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:47.765187 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:47.765241 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:47.767800 (+ 2559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.110.195:56547: BlockingRecv error: recv got EOF from 127.2.110.195:56547 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 26 06:15:48 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.193@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:15:48 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.193@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:15:48.502621 11693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:48.495653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34933 (local address 127.2.110.193:59527)
0926 06:15:48.495975 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:48.495999 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:48.496087 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:48.496440 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:48.496441 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:48.496623 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:48.496682 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:48.496684 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:48.497780 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:48.497781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:48.498389 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:48.498391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:48.498828 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:48.498829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:48.498893 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:48.499642 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:48.499647 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:48.502482 (+ 2835us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.110.254:34933: Server oryx/127.2.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":207,"threads_started":1}
W20260926 06:15:48.502662 11694 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:48.495879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:59527 (local address 127.2.110.254:34933)
0926 06:15:48.496073 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:48.496076 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:48.496091 (+ 15us) server_negotiation.cc:407] Connection header received
0926 06:15:48.496253 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:48.496255 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:48.496270 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:48.496382 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:48.496812 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:48.497663 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:48.498496 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:48.498722 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:48.498826 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:48.502584 (+ 3758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.110.193:59527: BlockingRecv error: recv got EOF from 127.2.110.193:59527 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":125,"threads_started":1}
Sep 26 06:15:48 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.194@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:15:48 dist-test-slave-hndd krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.110.194@KRBTEST.COM for oryx/127.2.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:15:48.655800 11695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:48.651105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:34933 (local address 127.2.110.194:38137)
0926 06:15:48.651226 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:48.651239 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:48.651326 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:48.651490 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:48.651491 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:48.651671 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:48.651716 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:48.651718 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:48.652404 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:48.652405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:48.652934 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:48.652935 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:48.653373 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:48.653373 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:48.653438 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:48.653862 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:48.653867 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:48.655738 (+ 1871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.110.254:34933: Server oryx/127.2.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":71,"threads_started":1}
W20260926 06:15:48.655933 11694 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:48.651172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:38137 (local address 127.2.110.254:34933)
0926 06:15:48.651230 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:48.651233 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:48.651300 (+ 67us) server_negotiation.cc:407] Connection header received
0926 06:15:48.651411 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:48.651412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:48.651424 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:48.651483 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:48.651798 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:48.652308 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:48.653022 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:48.653247 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:48.653329 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:48.655845 (+ 2516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.110.194:38137: BlockingRecv error: recv got EOF from 127.2.110.194:38137 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 06:15:48.668763 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 11219
I20260926 06:15:48.673802 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 11371
I20260926 06:15:48.678692 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 11503
I20260926 06:15:48.683055 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 11149
2026-09-26T06:15:48Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17497 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T06:15:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:15:48Z Disabled control of system clock
I20260926 06:15:48.713052 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:44469
--webserver_interface=127.2.110.254
--webserver_port=0
--builtin_ntp_servers=127.2.110.212:36933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.110.254:44469
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 06:15:48.804435 11699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:48.804682 11699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:48.804703 11699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:48.806023 11699 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:15:48.806092 11699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:48.806105 11699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:48.806118 11699 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:15:48.806131 11699 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:15:48.807529 11699 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:36933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.110.254:44469
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:44469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:48.807839 11699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:48.808122 11699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:48.814467 11704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:48.814455 11705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:48.814533 11699 server_base.cc:1034] running on GCE node
W20260926 06:15:48.814429 11707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:48.814972 11699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:48.815342 11699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:48.816520 11699 hybrid_clock.cc:648] HybridClock initialized: now 1790403348816493 us; error 43 us; skew 500 ppm
I20260926 06:15:48.818181 11699 webserver.cc:460] Webserver started at http://127.2.110.254:37305/ using document root <none> and password file <none>
I20260926 06:15:48.818529 11699 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:48.818593 11699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:48.818773 11699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:48.819911 11699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/master-0/data/instance:
uuid: "177573ae9e274249b35a76a6755fcb5f"
format_stamp: "Formatted at 2026-09-26 06:15:48 on dist-test-slave-hndd"
server_key: "6b08cfcbaeed301df11d38137e8f8eba"
server_key_iv: "1beee0a27d331d751f5af26476026eb6"
server_key_version: "encryptionkey@0"
I20260926 06:15:48.820288 11699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/master-0/wal/instance:
uuid: "177573ae9e274249b35a76a6755fcb5f"
format_stamp: "Formatted at 2026-09-26 06:15:48 on dist-test-slave-hndd"
server_key: "6b08cfcbaeed301df11d38137e8f8eba"
server_key_iv: "1beee0a27d331d751f5af26476026eb6"
server_key_version: "encryptionkey@0"
I20260926 06:15:48.821750 11699 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:15:48.822715 11713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:48.823228 11699 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:48.823310 11699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "177573ae9e274249b35a76a6755fcb5f"
format_stamp: "Formatted at 2026-09-26 06:15:48 on dist-test-slave-hndd"
server_key: "6b08cfcbaeed301df11d38137e8f8eba"
server_key_iv: "1beee0a27d331d751f5af26476026eb6"
server_key_version: "encryptionkey@0"
I20260926 06:15:48.823427 11699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:48.848709 11699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:48.849321 11699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:48.849658 11699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:48.855916 11699 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.254:44469
I20260926 06:15:48.855965 11765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.254:44469 every 8 connection(s)
I20260926 06:15:48.856537 11699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:15:48.857298 11766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:48.858736 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 11699
I20260926 06:15:48.858865 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/master-0/wal/instance
I20260926 06:15:48.859093 2491 external_mini_cluster.cc:1428] Setting key 4122e5e184c71a37db37123954a5a490
I20260926 06:15:48.860963 11766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 177573ae9e274249b35a76a6755fcb5f: Bootstrap starting.
I20260926 06:15:48.861717 11766 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 177573ae9e274249b35a76a6755fcb5f: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:48.862017 11766 log.cc:824] T 00000000000000000000000000000000 P 177573ae9e274249b35a76a6755fcb5f: Log is configured to *not* fsync() on all Append() calls
I20260926 06:15:48.862735 11766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 177573ae9e274249b35a76a6755fcb5f: No bootstrap required, opened a new log
I20260926 06:15:48.864840 11769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:48.859796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54184 (local address 127.2.110.254:44469)
0926 06:15:48.860028 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:48.860036 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:48.860061 (+ 25us) server_negotiation.cc:407] Connection header received
0926 06:15:48.860495 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:48.860502 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:48.860561 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:48.860722 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:48.860924 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:48.861970 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:48.862781 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:48.863262 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:48.863417 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:48.864169 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:48.864173 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:48.864173 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:15:48.864368 (+ 195us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:48.864401 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:15:48.864416 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:48.864495 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:48.864572 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 06:15:48.864660 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":96,"threads_started":1}
I20260926 06:15:48.865873 11766 raft_consensus.cc:348] T 00000000000000000000000000000000 P 177573ae9e274249b35a76a6755fcb5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "177573ae9e274249b35a76a6755fcb5f" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 44469 } }
I20260926 06:15:48.866046 11766 raft_consensus.cc:374] T 00000000000000000000000000000000 P 177573ae9e274249b35a76a6755fcb5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:48.866071 11766 raft_consensus.cc:729] T 00000000000000000000000000000000 P 177573ae9e274249b35a76a6755fcb5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 177573ae9e274249b35a76a6755fcb5f, State: Initialized, Role: FOLLOWER
I20260926 06:15:48.866216 11766 consensus_queue.cc:260] T 00000000000000000000000000000000 P 177573ae9e274249b35a76a6755fcb5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "177573ae9e274249b35a76a6755fcb5f" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 44469 } }
I20260926 06:15:48.866293 11766 raft_consensus.cc:388] T 00000000000000000000000000000000 P 177573ae9e274249b35a76a6755fcb5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:15:48.866349 11766 raft_consensus.cc:482] T 00000000000000000000000000000000 P 177573ae9e274249b35a76a6755fcb5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:15:48.866400 11766 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 177573ae9e274249b35a76a6755fcb5f [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:48.867029 11766 raft_consensus.cc:504] T 00000000000000000000000000000000 P 177573ae9e274249b35a76a6755fcb5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "177573ae9e274249b35a76a6755fcb5f" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 44469 } }
I20260926 06:15:48.867129 11766 leader_election.cc:302] T 00000000000000000000000000000000 P 177573ae9e274249b35a76a6755fcb5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 177573ae9e274249b35a76a6755fcb5f; no voters:
I20260926 06:15:48.867314 11766 leader_election.cc:288] T 00000000000000000000000000000000 P 177573ae9e274249b35a76a6755fcb5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:15:48.867508 11771 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 177573ae9e274249b35a76a6755fcb5f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:15:48.867575 11766 sys_catalog.cc:564] T 00000000000000000000000000000000 P 177573ae9e274249b35a76a6755fcb5f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:15:48.867801 11771 raft_consensus.cc:686] T 00000000000000000000000000000000 P 177573ae9e274249b35a76a6755fcb5f [term 1 LEADER]: Becoming Leader. State: Replica: 177573ae9e274249b35a76a6755fcb5f, State: Running, Role: LEADER
I20260926 06:15:48.867978 11771 consensus_queue.cc:237] T 00000000000000000000000000000000 P 177573ae9e274249b35a76a6755fcb5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "177573ae9e274249b35a76a6755fcb5f" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 44469 } }
I20260926 06:15:48.868496 11773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 177573ae9e274249b35a76a6755fcb5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 177573ae9e274249b35a76a6755fcb5f. Latest consensus state: current_term: 1 leader_uuid: "177573ae9e274249b35a76a6755fcb5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "177573ae9e274249b35a76a6755fcb5f" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 44469 } } }
I20260926 06:15:48.868691 11773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 177573ae9e274249b35a76a6755fcb5f [sys.catalog]: This master's current role is: LEADER
I20260926 06:15:48.868795 11772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 177573ae9e274249b35a76a6755fcb5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "177573ae9e274249b35a76a6755fcb5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "177573ae9e274249b35a76a6755fcb5f" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 44469 } } }
I20260926 06:15:48.868898 11772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 177573ae9e274249b35a76a6755fcb5f [sys.catalog]: This master's current role is: LEADER
I20260926 06:15:48.869001 11780 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:15:48.870121 11780 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:15:48.872308 11780 catalog_manager.cc:1347] Generated new cluster ID: 434c112e1d294ecc8c5c929f05536a3f
I20260926 06:15:48.872400 11780 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:15:48.899175 11780 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:15:48.899778 11780 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:15:48.905925 11780 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 177573ae9e274249b35a76a6755fcb5f: Generated new TSK 0
I20260926 06:15:48.906159 11780 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:15:48.911211 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.193:0
--local_ip_for_outbound_sockets=127.2.110.193
--webserver_interface=127.2.110.193
--webserver_port=0
--tserver_master_addrs=127.2.110.254:44469
--builtin_ntp_servers=127.2.110.212:36933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 06:15:49.007021 11790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:49.007265 11790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:49.007293 11790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:49.009073 11790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:49.009171 11790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:49.009388 11790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.193
I20260926 06:15:49.011770 11790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:36933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.110.193
--webserver_port=0
--tserver_master_addrs=127.2.110.254:44469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.193
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:49.012018 11790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:49.012303 11790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:49.017650 11798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:49.017633 11796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:49.017918 11790 server_base.cc:1034] running on GCE node
W20260926 06:15:49.017700 11795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:49.018210 11790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:49.018455 11790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:49.019661 11790 hybrid_clock.cc:648] HybridClock initialized: now 1790403349019645 us; error 50 us; skew 500 ppm
I20260926 06:15:49.021261 11790 webserver.cc:460] Webserver started at http://127.2.110.193:41501/ using document root <none> and password file <none>
I20260926 06:15:49.021719 11790 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:49.021807 11790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:49.021997 11790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:49.023612 11790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-0/data/instance:
uuid: "a29c044982434bf2a4ae324c7531c50d"
format_stamp: "Formatted at 2026-09-26 06:15:49 on dist-test-slave-hndd"
server_key: "87d0fd8383953a414c3d5721879b0b20"
server_key_iv: "5f054410a349c0b22125f625931e10f4"
server_key_version: "encryptionkey@0"
I20260926 06:15:49.024084 11790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-0/wal/instance:
uuid: "a29c044982434bf2a4ae324c7531c50d"
format_stamp: "Formatted at 2026-09-26 06:15:49 on dist-test-slave-hndd"
server_key: "87d0fd8383953a414c3d5721879b0b20"
server_key_iv: "5f054410a349c0b22125f625931e10f4"
server_key_version: "encryptionkey@0"
I20260926 06:15:49.026187 11790 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.003s
I20260926 06:15:49.027347 11804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:49.027702 11790 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:49.027806 11790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-0/wal
uuid: "a29c044982434bf2a4ae324c7531c50d"
format_stamp: "Formatted at 2026-09-26 06:15:49 on dist-test-slave-hndd"
server_key: "87d0fd8383953a414c3d5721879b0b20"
server_key_iv: "5f054410a349c0b22125f625931e10f4"
server_key_version: "encryptionkey@0"
I20260926 06:15:49.027908 11790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:49.037550 11790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:49.037886 11790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:49.038049 11790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:49.038288 11790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:15:49.038622 11790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:15:49.038681 11790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:49.038724 11790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:15:49.038753 11790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:49.045925 11790 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.193:37211
I20260926 06:15:49.046015 11917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.193:37211 every 8 connection(s)
I20260926 06:15:49.046442 11790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
I20260926 06:15:49.047751 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 11790
I20260926 06:15:49.047851 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-0/wal/instance
I20260926 06:15:49.048064 2491 external_mini_cluster.cc:1428] Setting key adfad7a9a9bf106b66177d0badb1210a
I20260926 06:15:49.049111 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.194:0
--local_ip_for_outbound_sockets=127.2.110.194
--webserver_interface=127.2.110.194
--webserver_port=0
--tserver_master_addrs=127.2.110.254:44469
--builtin_ntp_servers=127.2.110.212:36933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 06:15:49.052891 11769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:49.047279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:54023 (local address 127.2.110.254:44469)
0926 06:15:49.047351 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:49.047353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:49.047645 (+ 292us) server_negotiation.cc:407] Connection header received
0926 06:15:49.048391 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:49.048392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:49.048404 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:49.048450 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:49.049381 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:49.049904 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:49.051237 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:49.051538 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:49.051631 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:49.052624 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:49.052625 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:49.052625 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:15:49.052630 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:49.052647 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:15:49.052651 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:49.052698 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:49.052800 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 06:15:49.052823 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 06:15:49.053185 11920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:49.047260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:44469 (local address 127.2.110.193:54023)
0926 06:15:49.047459 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:49.047504 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:49.048273 (+ 769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:49.048520 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:49.048525 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:49.048535 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:49.049233 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:49.049242 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:49.050001 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:49.050002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:49.051007 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:49.051009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:49.051650 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:49.051651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:49.051851 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:49.052456 (+ 605us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:15:49.052456 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:49.052496 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:15:49.052527 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:49.052717 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:49.052722 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:49.052725 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:15:49.052796 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 06:15:49.052914 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":89,"threads_started":1}
I20260926 06:15:49.053814 11918 heartbeater.cc:347] Connected to a master server at 127.2.110.254:44469
I20260926 06:15:49.053929 11918 heartbeater.cc:464] Registering TS with master...
I20260926 06:15:49.054150 11918 heartbeater.cc:511] Master 127.2.110.254:44469 requested a full tablet report, sending...
I20260926 06:15:49.054684 11730 ts_manager.cc:194] Registered new tserver with Master: a29c044982434bf2a4ae324c7531c50d (127.2.110.193:37211)
I20260926 06:15:49.055320 11730 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.110.193:54023
W20260926 06:15:49.137526 11921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:49.137768 11921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:49.137799 11921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:49.139022 11921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:49.139070 11921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:49.139097 11921 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.194
I20260926 06:15:49.140477 11921 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:36933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.110.194
--webserver_port=0
--tserver_master_addrs=127.2.110.254:44469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.194
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:49.140745 11921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:49.141075 11921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:49.146332 11927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:49.146327 11926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:49.146370 11929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:49.146534 11921 server_base.cc:1034] running on GCE node
I20260926 06:15:49.146733 11921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:49.146952 11921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:49.148164 11921 hybrid_clock.cc:648] HybridClock initialized: now 1790403349148152 us; error 29 us; skew 500 ppm
I20260926 06:15:49.149617 11921 webserver.cc:460] Webserver started at http://127.2.110.194:42841/ using document root <none> and password file <none>
I20260926 06:15:49.149900 11921 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:49.149974 11921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:49.150133 11921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:49.151258 11921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-1/data/instance:
uuid: "f8502693e8b24bddbd6c951b4fe48e5c"
format_stamp: "Formatted at 2026-09-26 06:15:49 on dist-test-slave-hndd"
server_key: "374f21dca35de2b946ef6f949c5ead77"
server_key_iv: "93c677f93407f45c2ef3162a320f1ff4"
server_key_version: "encryptionkey@0"
I20260926 06:15:49.151664 11921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-1/wal/instance:
uuid: "f8502693e8b24bddbd6c951b4fe48e5c"
format_stamp: "Formatted at 2026-09-26 06:15:49 on dist-test-slave-hndd"
server_key: "374f21dca35de2b946ef6f949c5ead77"
server_key_iv: "93c677f93407f45c2ef3162a320f1ff4"
server_key_version: "encryptionkey@0"
I20260926 06:15:49.153829 11921 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20260926 06:15:49.154937 11935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:49.155202 11921 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:49.155303 11921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-1/wal
uuid: "f8502693e8b24bddbd6c951b4fe48e5c"
format_stamp: "Formatted at 2026-09-26 06:15:49 on dist-test-slave-hndd"
server_key: "374f21dca35de2b946ef6f949c5ead77"
server_key_iv: "93c677f93407f45c2ef3162a320f1ff4"
server_key_version: "encryptionkey@0"
I20260926 06:15:49.155373 11921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:49.164974 11921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:49.165325 11921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:49.165508 11921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:49.165699 11921 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:15:49.166040 11921 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:15:49.166074 11921 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:49.166097 11921 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:15:49.166149 11921 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:49.172127 11921 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.194:34759
I20260926 06:15:49.172192 12048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.194:34759 every 8 connection(s)
I20260926 06:15:49.172564 11921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
I20260926 06:15:49.175623 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 11921
I20260926 06:15:49.175751 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-1/wal/instance
I20260926 06:15:49.175937 2491 external_mini_cluster.cc:1428] Setting key 1d650bf68977c8936cc545beb674875d
I20260926 06:15:49.176959 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.195:0
--local_ip_for_outbound_sockets=127.2.110.195
--webserver_interface=127.2.110.195
--webserver_port=0
--tserver_master_addrs=127.2.110.254:44469
--builtin_ntp_servers=127.2.110.212:36933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 06:15:49.178356 11769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:49.173597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:58615 (local address 127.2.110.254:44469)
0926 06:15:49.173647 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:49.173650 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:49.173679 (+ 29us) server_negotiation.cc:407] Connection header received
0926 06:15:49.174245 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:49.174246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:49.174259 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:49.174315 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:49.175014 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:49.175537 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:49.176797 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:49.176986 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:49.177070 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:49.178071 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:49.178073 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:49.178073 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:15:49.178078 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:49.178091 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:15:49.178095 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:49.178149 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:49.178250 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 06:15:49.178279 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 06:15:49.178614 12051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:49.173316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:44469 (local address 127.2.110.194:58615)
0926 06:15:49.173559 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:49.173600 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:49.174125 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:49.174409 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:49.174412 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:49.174421 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:49.174914 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:49.174918 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:49.175643 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:49.175644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:49.176698 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:49.176700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:49.177090 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:49.177091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:49.177235 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:49.177902 (+ 667us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:15:49.177902 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:49.177931 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:15:49.177954 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:49.178155 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:49.178161 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:49.178164 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:15:49.178253 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 06:15:49.178362 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":99,"threads_started":1}
I20260926 06:15:49.179292 12049 heartbeater.cc:347] Connected to a master server at 127.2.110.254:44469
I20260926 06:15:49.179391 12049 heartbeater.cc:464] Registering TS with master...
I20260926 06:15:49.179549 12049 heartbeater.cc:511] Master 127.2.110.254:44469 requested a full tablet report, sending...
I20260926 06:15:49.179955 11730 ts_manager.cc:194] Registered new tserver with Master: f8502693e8b24bddbd6c951b4fe48e5c (127.2.110.194:34759)
I20260926 06:15:49.180341 11730 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.110.194:58615
W20260926 06:15:49.264444 12052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:49.264657 12052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:49.264680 12052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:49.265905 12052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:49.265950 12052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:49.265978 12052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.195
I20260926 06:15:49.267477 12052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:36933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.110.195
--webserver_port=0
--tserver_master_addrs=127.2.110.254:44469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.195
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:49.267741 12052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:49.268067 12052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:49.273118 12060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:49.273125 12058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:49.273173 12057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:49.273414 12052 server_base.cc:1034] running on GCE node
I20260926 06:15:49.273639 12052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:49.273904 12052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:49.275068 12052 hybrid_clock.cc:648] HybridClock initialized: now 1790403349275064 us; error 32 us; skew 500 ppm
I20260926 06:15:49.276428 12052 webserver.cc:460] Webserver started at http://127.2.110.195:34993/ using document root <none> and password file <none>
I20260926 06:15:49.276728 12052 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:49.276820 12052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:49.276990 12052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:49.278286 12052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-2/data/instance:
uuid: "e2684690765643cb8a080a433de6eacc"
format_stamp: "Formatted at 2026-09-26 06:15:49 on dist-test-slave-hndd"
server_key: "554a7eddec6250e914540a0110a7784e"
server_key_iv: "76e66ce7fd077d093fcc552c41c75f52"
server_key_version: "encryptionkey@0"
I20260926 06:15:49.278685 12052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-2/wal/instance:
uuid: "e2684690765643cb8a080a433de6eacc"
format_stamp: "Formatted at 2026-09-26 06:15:49 on dist-test-slave-hndd"
server_key: "554a7eddec6250e914540a0110a7784e"
server_key_iv: "76e66ce7fd077d093fcc552c41c75f52"
server_key_version: "encryptionkey@0"
I20260926 06:15:49.280148 12052 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:15:49.281250 12066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:49.281482 12052 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:49.281586 12052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-2/wal
uuid: "e2684690765643cb8a080a433de6eacc"
format_stamp: "Formatted at 2026-09-26 06:15:49 on dist-test-slave-hndd"
server_key: "554a7eddec6250e914540a0110a7784e"
server_key_iv: "76e66ce7fd077d093fcc552c41c75f52"
server_key_version: "encryptionkey@0"
I20260926 06:15:49.281682 12052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:49.299623 12052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:49.299986 12052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:49.300148 12052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:49.300419 12052 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:15:49.300782 12052 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:15:49.300841 12052 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:49.300885 12052 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:15:49.300909 12052 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:49.307006 12052 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.195:34703
I20260926 06:15:49.307083 12179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.195:34703 every 8 connection(s)
I20260926 06:15:49.307458 12052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
I20260926 06:15:49.312973 11769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:49.308322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:56309 (local address 127.2.110.254:44469)
0926 06:15:49.308385 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:49.308386 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:49.308637 (+ 251us) server_negotiation.cc:407] Connection header received
0926 06:15:49.309126 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:49.309127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:49.309137 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:49.309202 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:49.309913 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:49.310509 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:49.311501 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:49.311685 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:49.311749 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:49.312694 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:49.312696 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:49.312696 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:15:49.312701 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:49.312715 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:15:49.312719 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:49.312763 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:49.312879 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 06:15:49.312903 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 06:15:49.313127 12182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:49.308298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:44469 (local address 127.2.110.195:56309)
0926 06:15:49.308504 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:49.308548 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:49.309032 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:49.309253 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:49.309257 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:49.309264 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:49.309797 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:49.309802 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:49.310605 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:49.310606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:49.311405 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:49.311407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:49.311765 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:49.311765 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:49.311873 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:49.312556 (+ 683us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:15:49.312557 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:49.312586 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:15:49.312609 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:49.312775 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:49.312778 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:49.312781 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:15:49.312834 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 06:15:49.312920 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":103,"threads_started":1}
I20260926 06:15:49.313303 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 12052
I20260926 06:15:49.313422 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-2/wal/instance
I20260926 06:15:49.313652 2491 external_mini_cluster.cc:1428] Setting key 7f6054f7c6487ac33e7e202b3a8d5264
I20260926 06:15:49.313854 12180 heartbeater.cc:347] Connected to a master server at 127.2.110.254:44469
I20260926 06:15:49.313974 12180 heartbeater.cc:464] Registering TS with master...
I20260926 06:15:49.314175 12180 heartbeater.cc:511] Master 127.2.110.254:44469 requested a full tablet report, sending...
I20260926 06:15:49.314566 11730 ts_manager.cc:194] Registered new tserver with Master: e2684690765643cb8a080a433de6eacc (127.2.110.195:34703)
I20260926 06:15:49.315068 11730 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.110.195:56309
I20260926 06:15:49.326171 2491 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 06:15:49.328238 11769 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:49.327653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54198 (local address 127.2.110.254:44469)
0926 06:15:49.327726 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:49.327729 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:49.327862 (+ 133us) server_negotiation.cc:407] Connection header received
0926 06:15:49.327902 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:49.327903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:49.327912 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:49.327962 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:49.328132 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54198: BlockingRecv error: recv got EOF from 127.0.0.1:54198 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 06:15:49.328389 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 11790
I20260926 06:15:49.333462 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 11921
I20260926 06:15:49.338114 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 12052
I20260926 06:15:49.343539 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 11699
2026-09-26T06:15:49Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (660 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T06:15:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:15:49Z Disabled control of system clock
I20260926 06:15:49.372709 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:35867
--webserver_interface=127.2.110.254
--webserver_port=0
--builtin_ntp_servers=127.2.110.212:42087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.110.254:35867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 06:15:49.458562 12191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:49.458786 12191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:49.458811 12191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:49.460255 12191 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:15:49.460319 12191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:49.460333 12191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:49.460347 12191 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:15:49.460363 12191 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:15:49.461966 12191 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:42087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.110.254:35867
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:35867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:49.462486 12191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:49.462975 12191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:49.469616 12199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:49.469802 12197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:49.469637 12191 server_base.cc:1034] running on GCE node
W20260926 06:15:49.469619 12196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:49.470187 12191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:49.470605 12191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:49.471799 12191 hybrid_clock.cc:648] HybridClock initialized: now 1790403349471768 us; error 46 us; skew 500 ppm
I20260926 06:15:49.473462 12191 webserver.cc:460] Webserver started at http://127.2.110.254:35167/ using document root <none> and password file <none>
I20260926 06:15:49.473779 12191 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:49.473871 12191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:49.474014 12191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:49.475229 12191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/master-0/data/instance:
uuid: "e081e8b39e084a229e565a16c89eb802"
format_stamp: "Formatted at 2026-09-26 06:15:49 on dist-test-slave-hndd"
server_key: "796e2bf1b668de979f56498274444e59"
server_key_iv: "c8b9c4d64aee02874bc7d61429c3b848"
server_key_version: "encryptionkey@0"
I20260926 06:15:49.475540 12191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/master-0/wal/instance:
uuid: "e081e8b39e084a229e565a16c89eb802"
format_stamp: "Formatted at 2026-09-26 06:15:49 on dist-test-slave-hndd"
server_key: "796e2bf1b668de979f56498274444e59"
server_key_iv: "c8b9c4d64aee02874bc7d61429c3b848"
server_key_version: "encryptionkey@0"
I20260926 06:15:49.476812 12191 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:15:49.477855 12205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:49.478029 12191 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:49.478120 12191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "e081e8b39e084a229e565a16c89eb802"
format_stamp: "Formatted at 2026-09-26 06:15:49 on dist-test-slave-hndd"
server_key: "796e2bf1b668de979f56498274444e59"
server_key_iv: "c8b9c4d64aee02874bc7d61429c3b848"
server_key_version: "encryptionkey@0"
I20260926 06:15:49.478180 12191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:49.481559 12191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:49.481798 12191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:49.481897 12191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:49.486171 12191 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.254:35867
I20260926 06:15:49.486205 12257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.254:35867 every 8 connection(s)
I20260926 06:15:49.486550 12191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:15:49.487259 12258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:49.487902 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 12191
I20260926 06:15:49.487977 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/master-0/wal/instance
I20260926 06:15:49.488128 2491 external_mini_cluster.cc:1428] Setting key 534401db9c42f4bdb57c63a85e6e6473
I20260926 06:15:49.490952 12261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:49.488688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40926 (local address 127.2.110.254:35867)
0926 06:15:49.488884 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:49.488892 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:49.488918 (+ 26us) server_negotiation.cc:407] Connection header received
0926 06:15:49.489476 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:49.489484 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:49.489533 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:49.489652 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:49.490372 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:49.490376 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:49.490376 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:15:49.490380 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:49.490419 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:15:49.490430 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:49.490489 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:49.490587 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 06:15:49.490781 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":84,"threads_started":1}
I20260926 06:15:49.491360 12258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e081e8b39e084a229e565a16c89eb802: Bootstrap starting.
I20260926 06:15:49.492038 12258 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e081e8b39e084a229e565a16c89eb802: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:49.492324 12258 log.cc:824] T 00000000000000000000000000000000 P e081e8b39e084a229e565a16c89eb802: Log is configured to *not* fsync() on all Append() calls
I20260926 06:15:49.493052 12258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e081e8b39e084a229e565a16c89eb802: No bootstrap required, opened a new log
I20260926 06:15:49.494464 12258 raft_consensus.cc:348] T 00000000000000000000000000000000 P e081e8b39e084a229e565a16c89eb802 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e081e8b39e084a229e565a16c89eb802" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 35867 } }
I20260926 06:15:49.494625 12258 raft_consensus.cc:374] T 00000000000000000000000000000000 P e081e8b39e084a229e565a16c89eb802 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:49.494673 12258 raft_consensus.cc:729] T 00000000000000000000000000000000 P e081e8b39e084a229e565a16c89eb802 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e081e8b39e084a229e565a16c89eb802, State: Initialized, Role: FOLLOWER
I20260926 06:15:49.494829 12258 consensus_queue.cc:260] T 00000000000000000000000000000000 P e081e8b39e084a229e565a16c89eb802 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e081e8b39e084a229e565a16c89eb802" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 35867 } }
I20260926 06:15:49.494941 12258 raft_consensus.cc:388] T 00000000000000000000000000000000 P e081e8b39e084a229e565a16c89eb802 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:15:49.494985 12258 raft_consensus.cc:482] T 00000000000000000000000000000000 P e081e8b39e084a229e565a16c89eb802 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:15:49.495034 12258 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e081e8b39e084a229e565a16c89eb802 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:49.495632 12258 raft_consensus.cc:504] T 00000000000000000000000000000000 P e081e8b39e084a229e565a16c89eb802 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e081e8b39e084a229e565a16c89eb802" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 35867 } }
I20260926 06:15:49.495752 12258 leader_election.cc:302] T 00000000000000000000000000000000 P e081e8b39e084a229e565a16c89eb802 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e081e8b39e084a229e565a16c89eb802; no voters:
I20260926 06:15:49.495970 12258 leader_election.cc:288] T 00000000000000000000000000000000 P e081e8b39e084a229e565a16c89eb802 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:15:49.496268 12263 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e081e8b39e084a229e565a16c89eb802 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:15:49.496266 12258 sys_catalog.cc:564] T 00000000000000000000000000000000 P e081e8b39e084a229e565a16c89eb802 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:15:49.496457 12263 raft_consensus.cc:686] T 00000000000000000000000000000000 P e081e8b39e084a229e565a16c89eb802 [term 1 LEADER]: Becoming Leader. State: Replica: e081e8b39e084a229e565a16c89eb802, State: Running, Role: LEADER
I20260926 06:15:49.496541 12263 consensus_queue.cc:237] T 00000000000000000000000000000000 P e081e8b39e084a229e565a16c89eb802 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e081e8b39e084a229e565a16c89eb802" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 35867 } }
I20260926 06:15:49.496955 12264 sys_catalog.cc:455] T 00000000000000000000000000000000 P e081e8b39e084a229e565a16c89eb802 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e081e8b39e084a229e565a16c89eb802" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e081e8b39e084a229e565a16c89eb802" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 35867 } } }
I20260926 06:15:49.496913 12265 sys_catalog.cc:455] T 00000000000000000000000000000000 P e081e8b39e084a229e565a16c89eb802 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e081e8b39e084a229e565a16c89eb802. Latest consensus state: current_term: 1 leader_uuid: "e081e8b39e084a229e565a16c89eb802" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e081e8b39e084a229e565a16c89eb802" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 35867 } } }
I20260926 06:15:49.497102 12265 sys_catalog.cc:458] T 00000000000000000000000000000000 P e081e8b39e084a229e565a16c89eb802 [sys.catalog]: This master's current role is: LEADER
I20260926 06:15:49.497092 12264 sys_catalog.cc:458] T 00000000000000000000000000000000 P e081e8b39e084a229e565a16c89eb802 [sys.catalog]: This master's current role is: LEADER
I20260926 06:15:49.497373 12272 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:15:49.497910 12272 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:15:49.499521 12272 catalog_manager.cc:1347] Generated new cluster ID: 3df49cda8ec04caab6f2b886bfc3abf3
I20260926 06:15:49.499578 12272 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:15:49.518217 12272 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:15:49.518541 12272 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:15:49.535000 12272 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e081e8b39e084a229e565a16c89eb802: Generated new TSK 0
I20260926 06:15:49.535223 12272 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:15:49.538863 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.193:0
--local_ip_for_outbound_sockets=127.2.110.193
--webserver_interface=127.2.110.193
--webserver_port=0
--tserver_master_addrs=127.2.110.254:35867
--builtin_ntp_servers=127.2.110.212:42087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 06:15:49.634272 12282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:49.634584 12282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:49.634631 12282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:49.636234 12282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:49.636313 12282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:49.636368 12282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.193
I20260926 06:15:49.638288 12282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:42087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.110.193
--webserver_port=0
--tserver_master_addrs=127.2.110.254:35867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.193
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:49.638621 12282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:49.639005 12282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:49.644984 12288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:49.644965 12290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:49.644965 12287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:49.645159 12282 server_base.cc:1034] running on GCE node
I20260926 06:15:49.645781 12282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:49.646152 12282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:49.647347 12282 hybrid_clock.cc:648] HybridClock initialized: now 1790403349647321 us; error 48 us; skew 500 ppm
I20260926 06:15:49.648882 12282 webserver.cc:460] Webserver started at http://127.2.110.193:46739/ using document root <none> and password file <none>
I20260926 06:15:49.649151 12282 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:49.649242 12282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:49.649443 12282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:49.650755 12282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-0/data/instance:
uuid: "0cd873775ab54c129c7eff2125842876"
format_stamp: "Formatted at 2026-09-26 06:15:49 on dist-test-slave-hndd"
server_key: "cd6b4904f7d86a5e294a788defde62a4"
server_key_iv: "271e3bdc407def840d5fd11e7d982e15"
server_key_version: "encryptionkey@0"
I20260926 06:15:49.651173 12282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-0/wal/instance:
uuid: "0cd873775ab54c129c7eff2125842876"
format_stamp: "Formatted at 2026-09-26 06:15:49 on dist-test-slave-hndd"
server_key: "cd6b4904f7d86a5e294a788defde62a4"
server_key_iv: "271e3bdc407def840d5fd11e7d982e15"
server_key_version: "encryptionkey@0"
I20260926 06:15:49.652937 12282 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 06:15:49.653934 12296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:49.654137 12282 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 06:15:49.654249 12282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-0/wal
uuid: "0cd873775ab54c129c7eff2125842876"
format_stamp: "Formatted at 2026-09-26 06:15:49 on dist-test-slave-hndd"
server_key: "cd6b4904f7d86a5e294a788defde62a4"
server_key_iv: "271e3bdc407def840d5fd11e7d982e15"
server_key_version: "encryptionkey@0"
I20260926 06:15:49.654330 12282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:49.657845 12282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:49.658130 12282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:49.658278 12282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:49.658521 12282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:15:49.658839 12282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:15:49.658895 12282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:49.658941 12282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:15:49.658967 12282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:49.665316 12282 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.193:36639
I20260926 06:15:49.665400 12409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.193:36639 every 8 connection(s)
I20260926 06:15:49.665796 12282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
I20260926 06:15:49.669694 12261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:49.666678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:49313 (local address 127.2.110.254:35867)
0926 06:15:49.666790 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:49.666793 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:49.667148 (+ 355us) server_negotiation.cc:407] Connection header received
0926 06:15:49.667994 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:49.667996 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:49.668002 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:49.668065 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:49.669459 (+ 1394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:49.669460 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:49.669461 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:15:49.669465 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:49.669479 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:15:49.669484 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:49.669537 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:49.669622 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 06:15:49.669635 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 06:15:49.669884 12412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:49.666608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35867 (local address 127.2.110.193:49313)
0926 06:15:49.666916 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:49.666975 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:49.667843 (+ 868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:49.668144 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:49.668147 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:49.668157 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:49.669235 (+ 1078us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:15:49.669235 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:49.669266 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:15:49.669311 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:49.669550 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:49.669554 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:49.669557 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:15:49.669618 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 06:15:49.669673 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":95,"threads_started":1}
I20260926 06:15:49.670706 12410 heartbeater.cc:347] Connected to a master server at 127.2.110.254:35867
I20260926 06:15:49.670814 12410 heartbeater.cc:464] Registering TS with master...
I20260926 06:15:49.670989 12410 heartbeater.cc:511] Master 127.2.110.254:35867 requested a full tablet report, sending...
I20260926 06:15:49.671545 12222 ts_manager.cc:194] Registered new tserver with Master: 0cd873775ab54c129c7eff2125842876 (127.2.110.193:36639)
I20260926 06:15:49.674578 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 12282
I20260926 06:15:49.674655 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-0/wal/instance
I20260926 06:15:49.675048 2491 external_mini_cluster.cc:1428] Setting key e741632eddf24074036052a7c5f4488e
I20260926 06:15:49.676599 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.194:0
--local_ip_for_outbound_sockets=127.2.110.194
--webserver_interface=127.2.110.194
--webserver_port=0
--tserver_master_addrs=127.2.110.254:35867
--builtin_ntp_servers=127.2.110.212:42087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 06:15:49.769205 12413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:49.769536 12413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:49.769568 12413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:49.771499 12413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:49.771554 12413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:49.771585 12413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.194
I20260926 06:15:49.773269 12413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:42087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.110.194
--webserver_port=0
--tserver_master_addrs=127.2.110.254:35867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.194
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:49.773614 12413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:49.773957 12413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 06:15:49.779762 12413 server_base.cc:1034] running on GCE node
W20260926 06:15:49.779736 12421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:49.779722 12418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:49.779718 12419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:49.780230 12413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:49.780568 12413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:49.781738 12413 hybrid_clock.cc:648] HybridClock initialized: now 1790403349781720 us; error 29 us; skew 500 ppm
I20260926 06:15:49.783229 12413 webserver.cc:460] Webserver started at http://127.2.110.194:42513/ using document root <none> and password file <none>
I20260926 06:15:49.783594 12413 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:49.783682 12413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:49.783824 12413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:49.784931 12413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-1/data/instance:
uuid: "7b50ace475e54e42826adb5567137093"
format_stamp: "Formatted at 2026-09-26 06:15:49 on dist-test-slave-hndd"
server_key: "4ca88750fb9720d8db4244fb8ac6c18d"
server_key_iv: "59f5b5ae6a46d92683bcadc9f9cff6d1"
server_key_version: "encryptionkey@0"
I20260926 06:15:49.785310 12413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-1/wal/instance:
uuid: "7b50ace475e54e42826adb5567137093"
format_stamp: "Formatted at 2026-09-26 06:15:49 on dist-test-slave-hndd"
server_key: "4ca88750fb9720d8db4244fb8ac6c18d"
server_key_iv: "59f5b5ae6a46d92683bcadc9f9cff6d1"
server_key_version: "encryptionkey@0"
I20260926 06:15:49.787062 12413 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260926 06:15:49.787985 12427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:49.788343 12413 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:49.788506 12413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-1/wal
uuid: "7b50ace475e54e42826adb5567137093"
format_stamp: "Formatted at 2026-09-26 06:15:49 on dist-test-slave-hndd"
server_key: "4ca88750fb9720d8db4244fb8ac6c18d"
server_key_iv: "59f5b5ae6a46d92683bcadc9f9cff6d1"
server_key_version: "encryptionkey@0"
I20260926 06:15:49.788712 12413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:49.793043 12413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:49.793665 12413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:49.793859 12413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:49.794281 12413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:15:49.794760 12413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:15:49.794804 12413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:49.794831 12413 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:15:49.794843 12413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:49.802627 12413 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.194:38507
I20260926 06:15:49.802703 12540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.194:38507 every 8 connection(s)
I20260926 06:15:49.803051 12413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
I20260926 06:15:49.803248 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 12413
I20260926 06:15:49.803380 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-1/wal/instance
I20260926 06:15:49.803599 2491 external_mini_cluster.cc:1428] Setting key 6682ad7ad1bd0af2f1686ed1a0eceba7
I20260926 06:15:49.804692 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.195:0
--local_ip_for_outbound_sockets=127.2.110.195
--webserver_interface=127.2.110.195
--webserver_port=0
--tserver_master_addrs=127.2.110.254:35867
--builtin_ntp_servers=127.2.110.212:42087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 06:15:49.806903 12261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:49.803890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:58415 (local address 127.2.110.254:35867)
0926 06:15:49.803954 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:49.803956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:49.804248 (+ 292us) server_negotiation.cc:407] Connection header received
0926 06:15:49.805059 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:49.805061 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:49.805067 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:49.805126 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:49.806610 (+ 1484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:49.806611 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:49.806612 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:15:49.806618 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:49.806638 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:15:49.806644 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:49.806730 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:49.806779 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 06:15:49.806799 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 06:15:49.807057 12543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:49.803899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35867 (local address 127.2.110.194:58415)
0926 06:15:49.804094 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:49.804158 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:49.804889 (+ 731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:49.805218 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:49.805223 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:49.805233 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:49.806423 (+ 1190us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:15:49.806423 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:49.806458 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:15:49.806490 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:49.806724 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:49.806726 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:49.806729 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:15:49.806783 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 06:15:49.806836 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":100,"threads_started":1}
I20260926 06:15:49.807559 12541 heartbeater.cc:347] Connected to a master server at 127.2.110.254:35867
I20260926 06:15:49.807687 12541 heartbeater.cc:464] Registering TS with master...
I20260926 06:15:49.807893 12541 heartbeater.cc:511] Master 127.2.110.254:35867 requested a full tablet report, sending...
I20260926 06:15:49.808346 12222 ts_manager.cc:194] Registered new tserver with Master: 7b50ace475e54e42826adb5567137093 (127.2.110.194:38507)
W20260926 06:15:49.889974 12544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:49.890211 12544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:49.890271 12544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:49.891593 12544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:49.891657 12544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:49.891688 12544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.195
I20260926 06:15:49.893303 12544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:42087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.110.195
--webserver_port=0
--tserver_master_addrs=127.2.110.254:35867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.195
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:49.893635 12544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:49.893929 12544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:49.899674 12550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:49.899729 12552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:49.899947 12544 server_base.cc:1034] running on GCE node
W20260926 06:15:49.899993 12549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:49.900365 12544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:49.900656 12544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:49.901852 12544 hybrid_clock.cc:648] HybridClock initialized: now 1790403349901825 us; error 41 us; skew 500 ppm
I20260926 06:15:49.903376 12544 webserver.cc:460] Webserver started at http://127.2.110.195:36549/ using document root <none> and password file <none>
I20260926 06:15:49.903657 12544 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:49.903712 12544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:49.903854 12544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:49.904951 12544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-2/data/instance:
uuid: "7e6b4d1bdb054e9880c22b331d2e380d"
format_stamp: "Formatted at 2026-09-26 06:15:49 on dist-test-slave-hndd"
server_key: "bb495493c73de229ba1e54431581385c"
server_key_iv: "8f4c583634f6579827d46df3dc4be3f3"
server_key_version: "encryptionkey@0"
I20260926 06:15:49.905324 12544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-2/wal/instance:
uuid: "7e6b4d1bdb054e9880c22b331d2e380d"
format_stamp: "Formatted at 2026-09-26 06:15:49 on dist-test-slave-hndd"
server_key: "bb495493c73de229ba1e54431581385c"
server_key_iv: "8f4c583634f6579827d46df3dc4be3f3"
server_key_version: "encryptionkey@0"
I20260926 06:15:49.906999 12544 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:15:49.907874 12558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:49.908108 12544 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:15:49.908319 12544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-2/wal
uuid: "7e6b4d1bdb054e9880c22b331d2e380d"
format_stamp: "Formatted at 2026-09-26 06:15:49 on dist-test-slave-hndd"
server_key: "bb495493c73de229ba1e54431581385c"
server_key_iv: "8f4c583634f6579827d46df3dc4be3f3"
server_key_version: "encryptionkey@0"
I20260926 06:15:49.908541 12544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:49.912251 12544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:49.912544 12544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:49.912689 12544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:49.912897 12544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:15:49.913208 12544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:15:49.913264 12544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:49.913313 12544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:15:49.913369 12544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:49.919821 12544 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.195:44615
I20260926 06:15:49.919895 12671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.195:44615 every 8 connection(s)
I20260926 06:15:49.920271 12544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
I20260926 06:15:49.920924 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 12544
I20260926 06:15:49.921046 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403292369990-2491-0/minicluster-data/ts-2/wal/instance
I20260926 06:15:49.921283 2491 external_mini_cluster.cc:1428] Setting key 91637eb9ed17c80390347e693fab1276
I20260926 06:15:49.923743 12261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:49.921117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:38575 (local address 127.2.110.254:35867)
0926 06:15:49.921178 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:49.921181 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:49.921578 (+ 397us) server_negotiation.cc:407] Connection header received
0926 06:15:49.922209 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:49.922210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:49.922217 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:49.922290 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:49.923558 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:49.923559 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:49.923560 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:15:49.923564 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:49.923578 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:15:49.923582 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:49.923631 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:49.923679 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 06:15:49.923691 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 06:15:49.923946 12674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:49.921068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35867 (local address 127.2.110.195:38575)
0926 06:15:49.921289 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:49.921369 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:49.922044 (+ 675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:49.922403 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:49.922406 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:49.922414 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:49.923415 (+ 1001us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:15:49.923416 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:49.923444 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:15:49.923472 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:49.923630 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:49.923632 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:49.923634 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 06:15:49.923685 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 06:15:49.923733 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":91,"threads_started":1}
I20260926 06:15:49.924397 12672 heartbeater.cc:347] Connected to a master server at 127.2.110.254:35867
I20260926 06:15:49.924474 12672 heartbeater.cc:464] Registering TS with master...
I20260926 06:15:49.924646 12672 heartbeater.cc:511] Master 127.2.110.254:35867 requested a full tablet report, sending...
I20260926 06:15:49.925014 12222 ts_manager.cc:194] Registered new tserver with Master: 7e6b4d1bdb054e9880c22b331d2e380d (127.2.110.195:44615)
I20260926 06:15:49.933560 2491 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 06:15:49.935446 12261 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:15:49.934983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40928 (local address 127.2.110.254:35867)
0926 06:15:49.935040 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:49.935042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:49.935150 (+ 108us) server_negotiation.cc:407] Connection header received
0926 06:15:49.935197 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:49.935197 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:49.935202 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:49.935247 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:49.935362 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40928: BlockingRecv error: recv got EOF from 127.0.0.1:40928 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 06:15:49.935525 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 12282
I20260926 06:15:49.940176 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 12413
I20260926 06:15:49.945329 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 12544
I20260926 06:15:49.950327 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 12191
2026-09-26T06:15:49Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (607 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:15:50 dist-test-slave-hndd krb5kdc[12682](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:15:50 dist-test-slave-hndd krb5kdc[12682](info): set up 2 sockets
Sep 26 06:15:50 dist-test-slave-hndd krb5kdc[12682](info): commencing operation
krb5kdc: starting...
W20260926 06:15:51.026780 2491 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Sep 26 06:15:51 dist-test-slave-hndd krb5kdc[12682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:15:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:15:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
I20260926 06:15:51.153650 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:33271
--webserver_interface=127.2.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:46429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.110.254:33271
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:15:51.242290 12695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:51.242532 12695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:51.242556 12695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:51.243691 12695 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:15:51.243728 12695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:51.243743 12695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:51.243755 12695 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:15:51.243768 12695 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:15:51.245568 12695 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:46429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.110.254:33271
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:33271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:51.245793 12695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:51.245980 12695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:51.251415 12701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:51.251541 12700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:51.251703 12695 server_base.cc:1034] running on GCE node
W20260926 06:15:51.251541 12703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:51.252178 12695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:51.252533 12695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:51.253732 12695 hybrid_clock.cc:648] HybridClock initialized: now 1790403351253723 us; error 44 us; skew 500 ppm
Sep 26 06:15:51 dist-test-slave-hndd krb5kdc[12682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403351, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:51.256429 12695 init.cc:377] Logged in from keytab as kudu/127.2.110.254@KRBTEST.COM (short username kudu)
I20260926 06:15:51.256845 12695 webserver.cc:460] Webserver started at http://127.2.110.254:44891/ using document root <none> and password file <none>
I20260926 06:15:51.257047 12695 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:51.257088 12695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:51.257179 12695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:51.258304 12695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/master-0/data/instance:
uuid: "f93460d1ed69447d84af432f2202f951"
format_stamp: "Formatted at 2026-09-26 06:15:51 on dist-test-slave-hndd"
server_key: "9a0005c055f6ef4d572559e58ea55ea8"
server_key_iv: "23ad16af32c3572c384aa09acb818d33"
server_key_version: "encryptionkey@0"
I20260926 06:15:51.258610 12695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/master-0/wal/instance:
uuid: "f93460d1ed69447d84af432f2202f951"
format_stamp: "Formatted at 2026-09-26 06:15:51 on dist-test-slave-hndd"
server_key: "9a0005c055f6ef4d572559e58ea55ea8"
server_key_iv: "23ad16af32c3572c384aa09acb818d33"
server_key_version: "encryptionkey@0"
I20260926 06:15:51.260028 12695 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:15:51.260756 12710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:51.260929 12695 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:15:51.261020 12695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "f93460d1ed69447d84af432f2202f951"
format_stamp: "Formatted at 2026-09-26 06:15:51 on dist-test-slave-hndd"
server_key: "9a0005c055f6ef4d572559e58ea55ea8"
server_key_iv: "23ad16af32c3572c384aa09acb818d33"
server_key_version: "encryptionkey@0"
I20260926 06:15:51.261081 12695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:51.276731 12695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:51.279175 12695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:51.279300 12695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:51.283371 12695 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.254:33271
I20260926 06:15:51.283568 12762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.254:33271 every 8 connection(s)
I20260926 06:15:51.283839 12695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:15:51.285110 12763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:51.288305 12763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f93460d1ed69447d84af432f2202f951: Bootstrap starting.
I20260926 06:15:51.288983 12763 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f93460d1ed69447d84af432f2202f951: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:51.289280 12763 log.cc:824] T 00000000000000000000000000000000 P f93460d1ed69447d84af432f2202f951: Log is configured to *not* fsync() on all Append() calls
I20260926 06:15:51.289605 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 12695
I20260926 06:15:51.289721 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/master-0/wal/instance
I20260926 06:15:51.289924 2491 external_mini_cluster.cc:1428] Setting key b02a2fea7fdcc5677d0f73cfa48f7482
I20260926 06:15:51.290553 12763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f93460d1ed69447d84af432f2202f951: No bootstrap required, opened a new log
I20260926 06:15:51.293016 12763 raft_consensus.cc:348] T 00000000000000000000000000000000 P f93460d1ed69447d84af432f2202f951 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f93460d1ed69447d84af432f2202f951" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 33271 } }
I20260926 06:15:51.293229 12763 raft_consensus.cc:374] T 00000000000000000000000000000000 P f93460d1ed69447d84af432f2202f951 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:51.293267 12763 raft_consensus.cc:729] T 00000000000000000000000000000000 P f93460d1ed69447d84af432f2202f951 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f93460d1ed69447d84af432f2202f951, State: Initialized, Role: FOLLOWER
I20260926 06:15:51.293473 12763 consensus_queue.cc:260] T 00000000000000000000000000000000 P f93460d1ed69447d84af432f2202f951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f93460d1ed69447d84af432f2202f951" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 33271 } }
I20260926 06:15:51.293623 12763 raft_consensus.cc:388] T 00000000000000000000000000000000 P f93460d1ed69447d84af432f2202f951 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:15:51.293691 12763 raft_consensus.cc:482] T 00000000000000000000000000000000 P f93460d1ed69447d84af432f2202f951 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:15:51.293762 12763 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f93460d1ed69447d84af432f2202f951 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:51.294925 12763 raft_consensus.cc:504] T 00000000000000000000000000000000 P f93460d1ed69447d84af432f2202f951 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f93460d1ed69447d84af432f2202f951" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 33271 } }
I20260926 06:15:51.295033 12763 leader_election.cc:302] T 00000000000000000000000000000000 P f93460d1ed69447d84af432f2202f951 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f93460d1ed69447d84af432f2202f951; no voters:
I20260926 06:15:51.295308 12763 leader_election.cc:288] T 00000000000000000000000000000000 P f93460d1ed69447d84af432f2202f951 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:15:51.295329 12768 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f93460d1ed69447d84af432f2202f951 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:15:51.295447 12768 raft_consensus.cc:686] T 00000000000000000000000000000000 P f93460d1ed69447d84af432f2202f951 [term 1 LEADER]: Becoming Leader. State: Replica: f93460d1ed69447d84af432f2202f951, State: Running, Role: LEADER
I20260926 06:15:51.295604 12768 consensus_queue.cc:237] T 00000000000000000000000000000000 P f93460d1ed69447d84af432f2202f951 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f93460d1ed69447d84af432f2202f951" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 33271 } }
I20260926 06:15:51.295614 12763 sys_catalog.cc:564] T 00000000000000000000000000000000 P f93460d1ed69447d84af432f2202f951 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:15:51.296263 12770 sys_catalog.cc:455] T 00000000000000000000000000000000 P f93460d1ed69447d84af432f2202f951 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f93460d1ed69447d84af432f2202f951. Latest consensus state: current_term: 1 leader_uuid: "f93460d1ed69447d84af432f2202f951" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f93460d1ed69447d84af432f2202f951" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 33271 } } }
I20260926 06:15:51.296263 12769 sys_catalog.cc:455] T 00000000000000000000000000000000 P f93460d1ed69447d84af432f2202f951 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f93460d1ed69447d84af432f2202f951" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f93460d1ed69447d84af432f2202f951" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 33271 } } }
I20260926 06:15:51.296388 12770 sys_catalog.cc:458] T 00000000000000000000000000000000 P f93460d1ed69447d84af432f2202f951 [sys.catalog]: This master's current role is: LEADER
I20260926 06:15:51.296391 12769 sys_catalog.cc:458] T 00000000000000000000000000000000 P f93460d1ed69447d84af432f2202f951 [sys.catalog]: This master's current role is: LEADER
I20260926 06:15:51.297078 12773 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:15:51.297659 12773 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:15:51.299916 12773 catalog_manager.cc:1347] Generated new cluster ID: ad3ccc86d3ba4f82a0757f532376ca04
I20260926 06:15:51.299974 12773 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 06:15:51 dist-test-slave-hndd krb5kdc[12682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:15:51.305696 12767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:51.290625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55784 (local address 127.2.110.254:33271)
0926 06:15:51.290914 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:51.290924 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:51.290958 (+ 34us) server_negotiation.cc:407] Connection header received
0926 06:15:51.293046 (+ 2088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:51.293059 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:51.293143 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:51.293456 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:51.294324 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:51.295560 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:51.296816 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:51.297849 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:51.298017 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:51.301202 (+ 3185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:51.301206 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:51.301207 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:51.301219 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:51.303367 (+ 2148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:51.304085 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:51.304088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:51.304088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:51.304254 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:51.304541 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:51.304542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:51.304542 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:51.304888 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:51.305053 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:51.305275 (+ 222us) server_negotiation.cc:299] Negotiation successful
0926 06:15:51.305472 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":168,"threads_started":1}
I20260926 06:15:51.309265 12773 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:15:51.309880 12773 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:15:51.315953 12773 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f93460d1ed69447d84af432f2202f951: Generated new TSK 0
I20260926 06:15:51.316144 12773 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab.
Entry for principal kudu/127.2.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab.
I20260926 06:15:51.349921 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.193:0
--local_ip_for_outbound_sockets=127.2.110.193
--webserver_interface=127.2.110.193
--webserver_port=0
--tserver_master_addrs=127.2.110.254:33271
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab
--principal=kudu/127.2.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:46429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:15:51.441692 12789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:51.441990 12789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:51.442037 12789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:51.443357 12789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:51.443446 12789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:51.443498 12789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.193
I20260926 06:15:51.445169 12789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:46429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab
--principal=kudu/127.2.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.110.193
--webserver_port=0
--tserver_master_addrs=127.2.110.254:33271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.193
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:51.445510 12789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:51.445815 12789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:51.452533 12794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:51.452543 12795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:51.452564 12789 server_base.cc:1034] running on GCE node
W20260926 06:15:51.452533 12797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:51.453150 12789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:51.453492 12789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:51.454670 12789 hybrid_clock.cc:648] HybridClock initialized: now 1790403351454643 us; error 64 us; skew 500 ppm
Sep 26 06:15:51 dist-test-slave-hndd krb5kdc[12682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403351, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:51.457723 12789 init.cc:377] Logged in from keytab as kudu/127.2.110.193@KRBTEST.COM (short username kudu)
I20260926 06:15:51.458539 12789 webserver.cc:460] Webserver started at http://127.2.110.193:38945/ using document root <none> and password file <none>
I20260926 06:15:51.458828 12789 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:51.458909 12789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:51.459072 12789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:51.460182 12789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/ts-0/data/instance:
uuid: "fe6063bdf7ce481085ee6831656d3935"
format_stamp: "Formatted at 2026-09-26 06:15:51 on dist-test-slave-hndd"
server_key: "60658429a5dda76217673358c52a830f"
server_key_iv: "b758de71b504944c21716279b4ac3e88"
server_key_version: "encryptionkey@0"
I20260926 06:15:51.460582 12789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/ts-0/wal/instance:
uuid: "fe6063bdf7ce481085ee6831656d3935"
format_stamp: "Formatted at 2026-09-26 06:15:51 on dist-test-slave-hndd"
server_key: "60658429a5dda76217673358c52a830f"
server_key_iv: "b758de71b504944c21716279b4ac3e88"
server_key_version: "encryptionkey@0"
I20260926 06:15:51.462201 12789 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260926 06:15:51.463056 12804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:51.463254 12789 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:15:51.463366 12789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/ts-0/wal
uuid: "fe6063bdf7ce481085ee6831656d3935"
format_stamp: "Formatted at 2026-09-26 06:15:51 on dist-test-slave-hndd"
server_key: "60658429a5dda76217673358c52a830f"
server_key_iv: "b758de71b504944c21716279b4ac3e88"
server_key_version: "encryptionkey@0"
I20260926 06:15:51.463455 12789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:51.486851 12789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:51.489632 12789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:51.489759 12789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:51.489985 12789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:15:51.490326 12789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:15:51.490363 12789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:51.490410 12789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:15:51.490437 12789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:51.497627 12789 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.193:34165
I20260926 06:15:51.497680 12917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.193:34165 every 8 connection(s)
I20260926 06:15:51.498085 12789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
Sep 26 06:15:51 dist-test-slave-hndd krb5kdc[12682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403351, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.193@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:15:51.506814 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 12789
I20260926 06:15:51.506950 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/ts-0/wal/instance
I20260926 06:15:51.507175 2491 external_mini_cluster.cc:1428] Setting key 4a4fae038ff78d483d4d1972ef00a925
I20260926 06:15:51.508821 12767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:51.498933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:43677 (local address 127.2.110.254:33271)
0926 06:15:51.499008 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:51.499045 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:51.499336 (+ 291us) server_negotiation.cc:407] Connection header received
0926 06:15:51.499908 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:51.499909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:51.499926 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:51.499990 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:51.501022 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:51.501639 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:51.502711 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:51.502941 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:51.503025 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:51.505601 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:51.505603 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:51.505603 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:51.505614 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:51.507139 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:51.507650 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:51.507651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:51.507652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:51.507716 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:51.508101 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:51.508102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:51.508102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:51.508214 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:51.508270 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:51.508673 (+ 403us) server_negotiation.cc:299] Negotiation successful
0926 06:15:51.508753 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 06:15:51.509145 12920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:51.498882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:33271 (local address 127.2.110.193:43677)
0926 06:15:51.499113 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:51.499162 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:51.499786 (+ 624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:51.500070 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:51.500074 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:51.500696 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:51.500907 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:51.500912 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:51.501743 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:51.501744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:51.502594 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:51.502597 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:51.503035 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:51.503035 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:51.503170 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:51.503790 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:51.503799 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:51.505474 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:51.507235 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:51.507247 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:51.507250 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:51.507571 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:51.507822 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:51.507832 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:51.507832 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:51.508015 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:51.508291 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:51.508297 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:51.508494 (+ 197us) client_negotiation.cc:769] Sending connection context
0926 06:15:51.508626 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 06:15:51.508842 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":114,"threads_started":1}
I20260926 06:15:51.510057 12918 heartbeater.cc:347] Connected to a master server at 127.2.110.254:33271
I20260926 06:15:51.510182 12918 heartbeater.cc:464] Registering TS with master...
I20260926 06:15:51.510382 12918 heartbeater.cc:511] Master 127.2.110.254:33271 requested a full tablet report, sending...
I20260926 06:15:51.510974 12727 ts_manager.cc:194] Registered new tserver with Master: fe6063bdf7ce481085ee6831656d3935 (127.2.110.193:34165)
I20260926 06:15:51.511601 12727 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.193@KRBTEST.COM'} at 127.2.110.193:43677
WARNING: no policy specified for kudu/127.2.110.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab.
Entry for principal kudu/127.2.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab.
I20260926 06:15:51.536059 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.194:0
--local_ip_for_outbound_sockets=127.2.110.194
--webserver_interface=127.2.110.194
--webserver_port=0
--tserver_master_addrs=127.2.110.254:33271
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab
--principal=kudu/127.2.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:46429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:15:51.621869 12923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:51.622155 12923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:51.622200 12923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:51.623433 12923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:51.623515 12923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:51.623564 12923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.194
I20260926 06:15:51.625068 12923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:46429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab
--principal=kudu/127.2.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.110.194
--webserver_port=0
--tserver_master_addrs=127.2.110.254:33271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.194
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:51.625422 12923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:51.625674 12923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:51.631165 12929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:51.631313 12931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:51.631234 12928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:51.631372 12923 server_base.cc:1034] running on GCE node
I20260926 06:15:51.631661 12923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:51.631970 12923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:51.633194 12923 hybrid_clock.cc:648] HybridClock initialized: now 1790403351633164 us; error 60 us; skew 500 ppm
Sep 26 06:15:51 dist-test-slave-hndd krb5kdc[12682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403351, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:51.636471 12923 init.cc:377] Logged in from keytab as kudu/127.2.110.194@KRBTEST.COM (short username kudu)
I20260926 06:15:51.637014 12923 webserver.cc:460] Webserver started at http://127.2.110.194:40139/ using document root <none> and password file <none>
I20260926 06:15:51.637521 12923 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:51.637645 12923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:51.637830 12923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:51.639170 12923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/ts-1/data/instance:
uuid: "d5566ed2ede34420984e3fc58557454b"
format_stamp: "Formatted at 2026-09-26 06:15:51 on dist-test-slave-hndd"
server_key: "5489e7090011981348e8734b144c2864"
server_key_iv: "26d0b3d498b934c31199d6c192085b47"
server_key_version: "encryptionkey@0"
I20260926 06:15:51.639603 12923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/ts-1/wal/instance:
uuid: "d5566ed2ede34420984e3fc58557454b"
format_stamp: "Formatted at 2026-09-26 06:15:51 on dist-test-slave-hndd"
server_key: "5489e7090011981348e8734b144c2864"
server_key_iv: "26d0b3d498b934c31199d6c192085b47"
server_key_version: "encryptionkey@0"
I20260926 06:15:51.641009 12923 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:15:51.641867 12938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:51.642036 12923 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:51.642143 12923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/ts-1/wal
uuid: "d5566ed2ede34420984e3fc58557454b"
format_stamp: "Formatted at 2026-09-26 06:15:51 on dist-test-slave-hndd"
server_key: "5489e7090011981348e8734b144c2864"
server_key_iv: "26d0b3d498b934c31199d6c192085b47"
server_key_version: "encryptionkey@0"
I20260926 06:15:51.642228 12923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:51.654390 12923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:51.656389 12923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:51.656576 12923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:51.656847 12923 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:15:51.657167 12923 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:15:51.657222 12923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:51.657259 12923 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:15:51.657310 12923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:51.664391 12923 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.194:40521
I20260926 06:15:51.664438 13051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.194:40521 every 8 connection(s)
I20260926 06:15:51.664788 12923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
Sep 26 06:15:51 dist-test-slave-hndd krb5kdc[12682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403351, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.194@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:15:51.671587 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 12923
I20260926 06:15:51.671667 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/ts-1/wal/instance
I20260926 06:15:51.671862 2491 external_mini_cluster.cc:1428] Setting key 7ea3cd232a3bb23962c259613e66024e
I20260926 06:15:51.674898 12767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:51.665621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:52997 (local address 127.2.110.254:33271)
0926 06:15:51.665677 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:51.665680 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:51.665871 (+ 191us) server_negotiation.cc:407] Connection header received
0926 06:15:51.666485 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:51.666486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:51.666497 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:51.666550 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:51.667553 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:51.668085 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:51.669096 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:51.669303 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:51.669424 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:51.671779 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:51.671780 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:51.671780 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:51.671795 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:51.673389 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:51.673794 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:51.673795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:51.673796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:51.673858 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:51.674198 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:51.674199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:51.674200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:51.674351 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:51.674424 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:51.674687 (+ 263us) server_negotiation.cc:299] Negotiation successful
0926 06:15:51.674784 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 06:15:51.674996 13054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:51.665573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:33271 (local address 127.2.110.194:52997)
0926 06:15:51.665724 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:51.665775 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:51.666341 (+ 566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:51.666647 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:51.666657 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:51.667240 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:51.667444 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:51.667448 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:51.668192 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:51.668193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:51.668995 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:51.668998 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:51.669431 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:51.669431 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:51.669567 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:51.670133 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:51.670143 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:51.671600 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:51.673459 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:51.673467 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:51.673469 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:51.673711 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:51.673954 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:51.673956 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:51.673956 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:51.674124 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:51.674427 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:51.674432 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:51.674558 (+ 126us) client_negotiation.cc:769] Sending connection context
0926 06:15:51.674644 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 06:15:51.674782 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":75,"threads_started":1}
I20260926 06:15:51.675503 13052 heartbeater.cc:347] Connected to a master server at 127.2.110.254:33271
I20260926 06:15:51.675630 13052 heartbeater.cc:464] Registering TS with master...
I20260926 06:15:51.675813 13052 heartbeater.cc:511] Master 127.2.110.254:33271 requested a full tablet report, sending...
I20260926 06:15:51.676229 12727 ts_manager.cc:194] Registered new tserver with Master: d5566ed2ede34420984e3fc58557454b (127.2.110.194:40521)
I20260926 06:15:51.676662 12727 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.194@KRBTEST.COM'} at 127.2.110.194:52997
WARNING: no policy specified for kudu/127.2.110.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab.
Entry for principal kudu/127.2.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab.
I20260926 06:15:51.700023 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.195:0
--local_ip_for_outbound_sockets=127.2.110.195
--webserver_interface=127.2.110.195
--webserver_port=0
--tserver_master_addrs=127.2.110.254:33271
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab
--principal=kudu/127.2.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:46429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:15:51.789659 13057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:15:51.789942 13057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:15:51.789986 13057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:15:51.791353 13057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:15:51.791438 13057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:15:51.791487 13057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.195
I20260926 06:15:51.793934 13057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:46429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab
--principal=kudu/127.2.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.110.195
--webserver_port=0
--tserver_master_addrs=127.2.110.254:33271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.195
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:15:51.794273 13057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:15:51.794541 13057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:15:51.800843 13065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:51.800843 13062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:15:51.800900 13063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:15:51.801919 13057 server_base.cc:1034] running on GCE node
I20260926 06:15:51.802173 13057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:15:51.802464 13057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:15:51.803632 13057 hybrid_clock.cc:648] HybridClock initialized: now 1790403351803614 us; error 30 us; skew 500 ppm
Sep 26 06:15:51 dist-test-slave-hndd krb5kdc[12682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403351, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:15:51.806573 13057 init.cc:377] Logged in from keytab as kudu/127.2.110.195@KRBTEST.COM (short username kudu)
I20260926 06:15:51.807013 13057 webserver.cc:460] Webserver started at http://127.2.110.195:40491/ using document root <none> and password file <none>
I20260926 06:15:51.807287 13057 fs_manager.cc:359] Metadata directory not provided
I20260926 06:15:51.807337 13057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:15:51.807473 13057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:15:51.808604 13057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/ts-2/data/instance:
uuid: "39a9eef5cfb6464ab50e5ac2de33de1f"
format_stamp: "Formatted at 2026-09-26 06:15:51 on dist-test-slave-hndd"
server_key: "d86d945493e6455851850827e72be134"
server_key_iv: "e283ba47a43ba0f9692c98bba73b5024"
server_key_version: "encryptionkey@0"
I20260926 06:15:51.808987 13057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/ts-2/wal/instance:
uuid: "39a9eef5cfb6464ab50e5ac2de33de1f"
format_stamp: "Formatted at 2026-09-26 06:15:51 on dist-test-slave-hndd"
server_key: "d86d945493e6455851850827e72be134"
server_key_iv: "e283ba47a43ba0f9692c98bba73b5024"
server_key_version: "encryptionkey@0"
I20260926 06:15:51.811046 13057 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260926 06:15:51.812089 13072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:51.812419 13057 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:15:51.812518 13057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/ts-2/wal
uuid: "39a9eef5cfb6464ab50e5ac2de33de1f"
format_stamp: "Formatted at 2026-09-26 06:15:51 on dist-test-slave-hndd"
server_key: "d86d945493e6455851850827e72be134"
server_key_iv: "e283ba47a43ba0f9692c98bba73b5024"
server_key_version: "encryptionkey@0"
I20260926 06:15:51.812604 13057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:15:51.833046 13057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:15:51.835873 13057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:15:51.836246 13057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:15:51.836604 13057 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:15:51.836982 13057 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:15:51.837049 13057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:51.837101 13057 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:15:51.837128 13057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:15:51.844919 13057 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.195:42525
I20260926 06:15:51.845034 13185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.195:42525 every 8 connection(s)
I20260926 06:15:51.845546 13057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
I20260926 06:15:51.845948 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 13057
I20260926 06:15:51.846083 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/ts-2/wal/instance
I20260926 06:15:51.846292 2491 external_mini_cluster.cc:1428] Setting key f247be7eb9cc6f727baf220dcd01cb1e
Sep 26 06:15:51 dist-test-slave-hndd krb5kdc[12682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403351, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.195@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:15:51.856942 12767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:51.846782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:47343 (local address 127.2.110.254:33271)
0926 06:15:51.846847 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:51.846855 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:51.847201 (+ 346us) server_negotiation.cc:407] Connection header received
0926 06:15:51.848000 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:51.848001 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:51.848013 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:51.848074 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:51.849429 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:51.849948 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:51.851026 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:51.851226 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:51.851293 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:51.853775 (+ 2482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:51.853776 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:51.853777 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:51.853786 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:51.855329 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:51.855786 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:51.855787 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:51.855787 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:51.855844 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:51.856078 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:51.856078 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:51.856079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:51.856188 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:51.856242 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:51.856778 (+ 536us) server_negotiation.cc:299] Negotiation successful
0926 06:15:51.856868 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 06:15:51.857133 13188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:51.846758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:33271 (local address 127.2.110.195:47343)
0926 06:15:51.847006 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:51.847074 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:51.847902 (+ 828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:51.848140 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:51.848144 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:51.848896 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:15:51.849115 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:51.849121 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:51.850078 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:51.850079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:51.850920 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:51.850922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:51.851314 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:51.851315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:51.851429 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:51.852065 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:15:51.852076 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:15:51.853610 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:15:51.855425 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:51.855433 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:51.855435 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:51.855701 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:51.855910 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:15:51.855910 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:15:51.855910 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:15:51.856020 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:15:51.856305 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:15:51.856316 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:15:51.856604 (+ 288us) client_negotiation.cc:769] Sending connection context
0926 06:15:51.856728 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 06:15:51.856880 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":111,"threads_started":1}
I20260926 06:15:51.857892 13186 heartbeater.cc:347] Connected to a master server at 127.2.110.254:33271
I20260926 06:15:51.858017 13186 heartbeater.cc:464] Registering TS with master...
I20260926 06:15:51.858319 13186 heartbeater.cc:511] Master 127.2.110.254:33271 requested a full tablet report, sending...
I20260926 06:15:51.858839 12727 ts_manager.cc:194] Registered new tserver with Master: 39a9eef5cfb6464ab50e5ac2de33de1f (127.2.110.195:42525)
I20260926 06:15:51.859314 12727 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.195@KRBTEST.COM'} at 127.2.110.195:47343
I20260926 06:15:51.870620 2491 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 06:15:51.879002 12767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:51.872021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55800 (local address 127.2.110.254:33271)
0926 06:15:51.872111 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:51.872113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:51.872201 (+ 88us) server_negotiation.cc:407] Connection header received
0926 06:15:51.872283 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:51.872284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:51.872294 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:51.872354 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:15:51.872929 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:51.873514 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:51.874247 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:51.874464 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:51.874551 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:51.875831 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:15:51.875832 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:15:51.875833 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:15:51.875840 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:15:51.877568 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:51.878013 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:51.878014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:51.878014 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:51.878086 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:15:51.878311 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:15:51.878312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:15:51.878312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:15:51.878502 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:15:51.878618 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:51.878789 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 06:15:51.878886 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 06:15:51.881093 12727 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55800:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:15:51.882902 12727 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:15:51.890156 13199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:51.885369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.193:34165 (local address 127.0.0.1:51104)
0926 06:15:51.885740 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:51.885748 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:51.885785 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:51.886358 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:51.886359 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:51.886361 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:15:51.886404 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:51.886406 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:51.887476 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:51.887477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:51.889031 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:51.889033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:51.889834 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:51.889836 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:51.889919 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:51.889933 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 06:15:51.890007 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 06:15:51.890066 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"mutex_wait_us":32,"thread_start_us":97,"threads_started":1}
I20260926 06:15:51.890165 13201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:51.885618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51104 (local address 127.2.110.193:34165)
0926 06:15:51.886175 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:51.886180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:51.886191 (+ 11us) server_negotiation.cc:407] Connection header received
0926 06:15:51.886234 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:51.886236 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:51.886253 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:51.886340 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:15:51.886577 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:51.887356 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:51.889110 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:51.889730 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:51.889831 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:51.889903 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:51.890028 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 06:15:51.890072 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":71,"threads_started":1}
I20260926 06:15:51.890625 13197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:51.885309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.194:40521 (local address 127.0.0.1:44130)
0926 06:15:51.885509 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:51.885550 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:51.885620 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:51.885991 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:51.885994 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:51.885996 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:15:51.886110 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:51.886112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:51.887667 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:51.887668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:51.888909 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:51.888912 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:51.890380 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:51.890381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:51.890435 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:51.890439 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:15:51.890490 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 06:15:51.890536 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":2,"thread_start_us":89,"threads_started":1}
I20260926 06:15:51.890813 13202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:51.885607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45796 (local address 127.2.110.195:42525)
0926 06:15:51.886478 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:51.886483 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:51.886492 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:15:51.886531 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:51.886533 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:51.886550 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:51.886638 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:15:51.886765 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:51.887864 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:51.888935 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:51.890042 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:51.890124 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:51.890192 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:51.890667 (+ 475us) server_negotiation.cc:299] Negotiation successful
0926 06:15:51.890728 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":835,"thread_start_us":84,"threads_started":1}
I20260926 06:15:51.890946 13200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:51.885439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44130 (local address 127.2.110.194:40521)
0926 06:15:51.885809 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:51.885814 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:51.885824 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:15:51.885862 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:51.885864 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:51.885878 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:51.886013 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:15:51.886272 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:51.887548 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:51.888997 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:51.890042 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:51.890124 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:51.890192 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:51.890786 (+ 594us) server_negotiation.cc:299] Negotiation successful
0926 06:15:51.890857 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":97,"threads_started":1}
I20260926 06:15:51.890990 13198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:51.885346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.195:42525 (local address 127.0.0.1:45796)
0926 06:15:51.885617 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:51.885628 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:51.885672 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:51.886634 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:51.886635 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:51.886638 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:15:51.886682 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:51.886683 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:51.887981 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:51.887982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:51.888803 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:51.888805 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:51.890235 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:51.890236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:51.890540 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:51.890543 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:15:51.890591 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 06:15:51.890624 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":1,"thread_start_us":118,"threads_started":1}
I20260926 06:15:51.890980 12852 tablet_service.cc:1467] Processing CreateTablet for tablet 41beca9c5cef4e26876356bb647b9d91 (DEFAULT_TABLE table=test-table [id=2a82035b29324831b3a31a5aa41157e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:15:51.891379 12852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41beca9c5cef4e26876356bb647b9d91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:51.891678 13120 tablet_service.cc:1467] Processing CreateTablet for tablet 41beca9c5cef4e26876356bb647b9d91 (DEFAULT_TABLE table=test-table [id=2a82035b29324831b3a31a5aa41157e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:15:51.891976 13120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41beca9c5cef4e26876356bb647b9d91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:51.894497 13203 tablet_bootstrap.cc:492] T 41beca9c5cef4e26876356bb647b9d91 P fe6063bdf7ce481085ee6831656d3935: Bootstrap starting.
I20260926 06:15:51.895197 13203 tablet_bootstrap.cc:654] T 41beca9c5cef4e26876356bb647b9d91 P fe6063bdf7ce481085ee6831656d3935: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:51.895448 12986 tablet_service.cc:1467] Processing CreateTablet for tablet 41beca9c5cef4e26876356bb647b9d91 (DEFAULT_TABLE table=test-table [id=2a82035b29324831b3a31a5aa41157e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:15:51.895782 12986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41beca9c5cef4e26876356bb647b9d91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:15:51.895886 13203 log.cc:824] T 41beca9c5cef4e26876356bb647b9d91 P fe6063bdf7ce481085ee6831656d3935: Log is configured to *not* fsync() on all Append() calls
I20260926 06:15:51.896749 13205 tablet_bootstrap.cc:492] T 41beca9c5cef4e26876356bb647b9d91 P 39a9eef5cfb6464ab50e5ac2de33de1f: Bootstrap starting.
I20260926 06:15:51.896781 13203 tablet_bootstrap.cc:492] T 41beca9c5cef4e26876356bb647b9d91 P fe6063bdf7ce481085ee6831656d3935: No bootstrap required, opened a new log
I20260926 06:15:51.896840 13203 ts_tablet_manager.cc:1397] T 41beca9c5cef4e26876356bb647b9d91 P fe6063bdf7ce481085ee6831656d3935: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:15:51.897619 13205 tablet_bootstrap.cc:654] T 41beca9c5cef4e26876356bb647b9d91 P 39a9eef5cfb6464ab50e5ac2de33de1f: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:51.898226 13205 log.cc:824] T 41beca9c5cef4e26876356bb647b9d91 P 39a9eef5cfb6464ab50e5ac2de33de1f: Log is configured to *not* fsync() on all Append() calls
I20260926 06:15:51.898931 13207 tablet_bootstrap.cc:492] T 41beca9c5cef4e26876356bb647b9d91 P d5566ed2ede34420984e3fc58557454b: Bootstrap starting.
I20260926 06:15:51.899085 13203 raft_consensus.cc:348] T 41beca9c5cef4e26876356bb647b9d91 P fe6063bdf7ce481085ee6831656d3935 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5566ed2ede34420984e3fc58557454b" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 40521 } } peers { permanent_uuid: "39a9eef5cfb6464ab50e5ac2de33de1f" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 42525 } } peers { permanent_uuid: "fe6063bdf7ce481085ee6831656d3935" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 34165 } }
I20260926 06:15:51.899288 13203 raft_consensus.cc:374] T 41beca9c5cef4e26876356bb647b9d91 P fe6063bdf7ce481085ee6831656d3935 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:51.899326 13203 raft_consensus.cc:729] T 41beca9c5cef4e26876356bb647b9d91 P fe6063bdf7ce481085ee6831656d3935 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe6063bdf7ce481085ee6831656d3935, State: Initialized, Role: FOLLOWER
I20260926 06:15:51.899626 13203 consensus_queue.cc:260] T 41beca9c5cef4e26876356bb647b9d91 P fe6063bdf7ce481085ee6831656d3935 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5566ed2ede34420984e3fc58557454b" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 40521 } } peers { permanent_uuid: "39a9eef5cfb6464ab50e5ac2de33de1f" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 42525 } } peers { permanent_uuid: "fe6063bdf7ce481085ee6831656d3935" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 34165 } }
I20260926 06:15:51.899741 13207 tablet_bootstrap.cc:654] T 41beca9c5cef4e26876356bb647b9d91 P d5566ed2ede34420984e3fc58557454b: Neither blocks nor log segments found. Creating new log.
I20260926 06:15:51.899978 13203 ts_tablet_manager.cc:1428] T 41beca9c5cef4e26876356bb647b9d91 P fe6063bdf7ce481085ee6831656d3935: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 06:15:51.899998 12918 heartbeater.cc:503] Master 127.2.110.254:33271 was elected leader, sending a full tablet report...
I20260926 06:15:51.900414 13207 log.cc:824] T 41beca9c5cef4e26876356bb647b9d91 P d5566ed2ede34420984e3fc58557454b: Log is configured to *not* fsync() on all Append() calls
I20260926 06:15:51.900733 13205 tablet_bootstrap.cc:492] T 41beca9c5cef4e26876356bb647b9d91 P 39a9eef5cfb6464ab50e5ac2de33de1f: No bootstrap required, opened a new log
I20260926 06:15:51.900822 13205 ts_tablet_manager.cc:1397] T 41beca9c5cef4e26876356bb647b9d91 P 39a9eef5cfb6464ab50e5ac2de33de1f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 06:15:51.901458 13207 tablet_bootstrap.cc:492] T 41beca9c5cef4e26876356bb647b9d91 P d5566ed2ede34420984e3fc58557454b: No bootstrap required, opened a new log
I20260926 06:15:51.901530 13207 ts_tablet_manager.cc:1397] T 41beca9c5cef4e26876356bb647b9d91 P d5566ed2ede34420984e3fc58557454b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 06:15:51.903228 13205 raft_consensus.cc:348] T 41beca9c5cef4e26876356bb647b9d91 P 39a9eef5cfb6464ab50e5ac2de33de1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5566ed2ede34420984e3fc58557454b" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 40521 } } peers { permanent_uuid: "39a9eef5cfb6464ab50e5ac2de33de1f" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 42525 } } peers { permanent_uuid: "fe6063bdf7ce481085ee6831656d3935" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 34165 } }
I20260926 06:15:51.903474 13205 raft_consensus.cc:374] T 41beca9c5cef4e26876356bb647b9d91 P 39a9eef5cfb6464ab50e5ac2de33de1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:51.903515 13205 raft_consensus.cc:729] T 41beca9c5cef4e26876356bb647b9d91 P 39a9eef5cfb6464ab50e5ac2de33de1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39a9eef5cfb6464ab50e5ac2de33de1f, State: Initialized, Role: FOLLOWER
I20260926 06:15:51.903643 13205 consensus_queue.cc:260] T 41beca9c5cef4e26876356bb647b9d91 P 39a9eef5cfb6464ab50e5ac2de33de1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5566ed2ede34420984e3fc58557454b" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 40521 } } peers { permanent_uuid: "39a9eef5cfb6464ab50e5ac2de33de1f" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 42525 } } peers { permanent_uuid: "fe6063bdf7ce481085ee6831656d3935" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 34165 } }
I20260926 06:15:51.903679 13207 raft_consensus.cc:348] T 41beca9c5cef4e26876356bb647b9d91 P d5566ed2ede34420984e3fc58557454b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5566ed2ede34420984e3fc58557454b" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 40521 } } peers { permanent_uuid: "39a9eef5cfb6464ab50e5ac2de33de1f" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 42525 } } peers { permanent_uuid: "fe6063bdf7ce481085ee6831656d3935" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 34165 } }
I20260926 06:15:51.903858 13207 raft_consensus.cc:374] T 41beca9c5cef4e26876356bb647b9d91 P d5566ed2ede34420984e3fc58557454b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:15:51.903906 13207 raft_consensus.cc:729] T 41beca9c5cef4e26876356bb647b9d91 P d5566ed2ede34420984e3fc58557454b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5566ed2ede34420984e3fc58557454b, State: Initialized, Role: FOLLOWER
I20260926 06:15:51.903975 13205 ts_tablet_manager.cc:1428] T 41beca9c5cef4e26876356bb647b9d91 P 39a9eef5cfb6464ab50e5ac2de33de1f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 06:15:51.903985 13186 heartbeater.cc:503] Master 127.2.110.254:33271 was elected leader, sending a full tablet report...
I20260926 06:15:51.904030 13207 consensus_queue.cc:260] T 41beca9c5cef4e26876356bb647b9d91 P d5566ed2ede34420984e3fc58557454b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5566ed2ede34420984e3fc58557454b" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 40521 } } peers { permanent_uuid: "39a9eef5cfb6464ab50e5ac2de33de1f" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 42525 } } peers { permanent_uuid: "fe6063bdf7ce481085ee6831656d3935" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 34165 } }
I20260926 06:15:51.904295 13207 ts_tablet_manager.cc:1428] T 41beca9c5cef4e26876356bb647b9d91 P d5566ed2ede34420984e3fc58557454b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 06:15:51.904366 13052 heartbeater.cc:503] Master 127.2.110.254:33271 was elected leader, sending a full tablet report...
W20260926 06:15:51.915433 13053 tablet.cc:2369] T 41beca9c5cef4e26876356bb647b9d91 P d5566ed2ede34420984e3fc58557454b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:15:51.978255 13210 raft_consensus.cc:482] T 41beca9c5cef4e26876356bb647b9d91 P 39a9eef5cfb6464ab50e5ac2de33de1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:15:51.978427 13210 raft_consensus.cc:504] T 41beca9c5cef4e26876356bb647b9d91 P 39a9eef5cfb6464ab50e5ac2de33de1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5566ed2ede34420984e3fc58557454b" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 40521 } } peers { permanent_uuid: "39a9eef5cfb6464ab50e5ac2de33de1f" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 42525 } } peers { permanent_uuid: "fe6063bdf7ce481085ee6831656d3935" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 34165 } }
I20260926 06:15:51.978855 13210 leader_election.cc:288] T 41beca9c5cef4e26876356bb647b9d91 P 39a9eef5cfb6464ab50e5ac2de33de1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5566ed2ede34420984e3fc58557454b (127.2.110.194:40521), fe6063bdf7ce481085ee6831656d3935 (127.2.110.193:34165)
I20260926 06:15:51.983678 13201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:51.979228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:55901 (local address 127.2.110.193:34165)
0926 06:15:51.979270 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:51.979272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:51.979278 (+ 6us) server_negotiation.cc:407] Connection header received
0926 06:15:51.979314 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:51.979314 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:51.979345 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:51.979399 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:15:51.979668 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:51.980445 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:51.982396 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:51.983229 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:51.983340 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:51.983352 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:51.983543 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 06:15:51.983596 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 06:15:51.983678 13200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:51.979138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:37555 (local address 127.2.110.194:40521)
0926 06:15:51.979190 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:51.979192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:51.979407 (+ 215us) server_negotiation.cc:407] Connection header received
0926 06:15:51.979511 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:51.979512 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:51.979521 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:51.979572 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:15:51.979751 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:51.980451 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:51.982396 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:51.983230 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:51.983341 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:51.983352 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:51.983546 (+ 194us) server_negotiation.cc:299] Negotiation successful
0926 06:15:51.983596 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 06:15:51.983686 13188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:51.979049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.193:34165 (local address 127.2.110.195:55901)
0926 06:15:51.979151 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:51.979163 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:51.979204 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:51.979463 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:51.979464 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:51.979466 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:15:51.979551 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:51.979552 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:51.980565 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:51.980566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:51.982180 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:51.982184 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:51.983357 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:51.983358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:51.983451 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:51.983453 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 06:15:51.983518 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 06:15:51.983573 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"mutex_wait_us":95}
I20260926 06:15:51.983696 13212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:51.979070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.194:40521 (local address 127.2.110.195:37555)
0926 06:15:51.979297 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:51.979308 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:15:51.979416 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:15:51.979607 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:15:51.979608 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:15:51.979610 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:15:51.979675 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:51.979677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:51.980565 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:51.980566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:51.982180 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:15:51.982184 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:15:51.983357 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:15:51.983358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:15:51.983446 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:51.983451 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 06:15:51.983518 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 06:15:51.983573 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":112,"thread_start_us":148,"threads_started":1}
I20260926 06:15:51.984163 12872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41beca9c5cef4e26876356bb647b9d91" candidate_uuid: "39a9eef5cfb6464ab50e5ac2de33de1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe6063bdf7ce481085ee6831656d3935" is_pre_election: true
I20260926 06:15:51.984163 13006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41beca9c5cef4e26876356bb647b9d91" candidate_uuid: "39a9eef5cfb6464ab50e5ac2de33de1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5566ed2ede34420984e3fc58557454b" is_pre_election: true
I20260926 06:15:51.984367 13006 raft_consensus.cc:2454] T 41beca9c5cef4e26876356bb647b9d91 P d5566ed2ede34420984e3fc58557454b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39a9eef5cfb6464ab50e5ac2de33de1f in term 0.
I20260926 06:15:51.984367 12872 raft_consensus.cc:2454] T 41beca9c5cef4e26876356bb647b9d91 P fe6063bdf7ce481085ee6831656d3935 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39a9eef5cfb6464ab50e5ac2de33de1f in term 0.
I20260926 06:15:51.984654 13074 leader_election.cc:302] T 41beca9c5cef4e26876356bb647b9d91 P 39a9eef5cfb6464ab50e5ac2de33de1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39a9eef5cfb6464ab50e5ac2de33de1f, d5566ed2ede34420984e3fc58557454b; no voters:
I20260926 06:15:51.984932 13210 raft_consensus.cc:2781] T 41beca9c5cef4e26876356bb647b9d91 P 39a9eef5cfb6464ab50e5ac2de33de1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:15:51.985025 13210 raft_consensus.cc:482] T 41beca9c5cef4e26876356bb647b9d91 P 39a9eef5cfb6464ab50e5ac2de33de1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:15:51.985059 13210 raft_consensus.cc:3037] T 41beca9c5cef4e26876356bb647b9d91 P 39a9eef5cfb6464ab50e5ac2de33de1f [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:51.985848 13210 raft_consensus.cc:504] T 41beca9c5cef4e26876356bb647b9d91 P 39a9eef5cfb6464ab50e5ac2de33de1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5566ed2ede34420984e3fc58557454b" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 40521 } } peers { permanent_uuid: "39a9eef5cfb6464ab50e5ac2de33de1f" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 42525 } } peers { permanent_uuid: "fe6063bdf7ce481085ee6831656d3935" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 34165 } }
I20260926 06:15:51.986063 13210 leader_election.cc:288] T 41beca9c5cef4e26876356bb647b9d91 P 39a9eef5cfb6464ab50e5ac2de33de1f [CANDIDATE]: Term 1 election: Requested vote from peers d5566ed2ede34420984e3fc58557454b (127.2.110.194:40521), fe6063bdf7ce481085ee6831656d3935 (127.2.110.193:34165)
I20260926 06:15:51.986272 12872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41beca9c5cef4e26876356bb647b9d91" candidate_uuid: "39a9eef5cfb6464ab50e5ac2de33de1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe6063bdf7ce481085ee6831656d3935"
I20260926 06:15:51.986302 13006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41beca9c5cef4e26876356bb647b9d91" candidate_uuid: "39a9eef5cfb6464ab50e5ac2de33de1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5566ed2ede34420984e3fc58557454b"
I20260926 06:15:51.986388 12872 raft_consensus.cc:3037] T 41beca9c5cef4e26876356bb647b9d91 P fe6063bdf7ce481085ee6831656d3935 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:51.986418 13006 raft_consensus.cc:3037] T 41beca9c5cef4e26876356bb647b9d91 P d5566ed2ede34420984e3fc58557454b [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:15:51.987231 12872 raft_consensus.cc:2454] T 41beca9c5cef4e26876356bb647b9d91 P fe6063bdf7ce481085ee6831656d3935 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39a9eef5cfb6464ab50e5ac2de33de1f in term 1.
I20260926 06:15:51.987361 13006 raft_consensus.cc:2454] T 41beca9c5cef4e26876356bb647b9d91 P d5566ed2ede34420984e3fc58557454b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39a9eef5cfb6464ab50e5ac2de33de1f in term 1.
I20260926 06:15:51.987463 13075 leader_election.cc:302] T 41beca9c5cef4e26876356bb647b9d91 P 39a9eef5cfb6464ab50e5ac2de33de1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39a9eef5cfb6464ab50e5ac2de33de1f, fe6063bdf7ce481085ee6831656d3935; no voters:
I20260926 06:15:51.987586 13210 raft_consensus.cc:2781] T 41beca9c5cef4e26876356bb647b9d91 P 39a9eef5cfb6464ab50e5ac2de33de1f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:15:51.987772 13210 raft_consensus.cc:686] T 41beca9c5cef4e26876356bb647b9d91 P 39a9eef5cfb6464ab50e5ac2de33de1f [term 1 LEADER]: Becoming Leader. State: Replica: 39a9eef5cfb6464ab50e5ac2de33de1f, State: Running, Role: LEADER
I20260926 06:15:51.987901 13210 consensus_queue.cc:237] T 41beca9c5cef4e26876356bb647b9d91 P 39a9eef5cfb6464ab50e5ac2de33de1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5566ed2ede34420984e3fc58557454b" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 40521 } } peers { permanent_uuid: "39a9eef5cfb6464ab50e5ac2de33de1f" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 42525 } } peers { permanent_uuid: "fe6063bdf7ce481085ee6831656d3935" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 34165 } }
I20260926 06:15:51.988917 12727 catalog_manager.cc:5495] T 41beca9c5cef4e26876356bb647b9d91 P 39a9eef5cfb6464ab50e5ac2de33de1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 39a9eef5cfb6464ab50e5ac2de33de1f (127.2.110.195). New cstate: current_term: 1 leader_uuid: "39a9eef5cfb6464ab50e5ac2de33de1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5566ed2ede34420984e3fc58557454b" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 40521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39a9eef5cfb6464ab50e5ac2de33de1f" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 42525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe6063bdf7ce481085ee6831656d3935" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 34165 } health_report { overall_health: UNKNOWN } } }
W20260926 06:15:51.998953 12919 tablet.cc:2369] T 41beca9c5cef4e26876356bb647b9d91 P fe6063bdf7ce481085ee6831656d3935: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:15:52.007503 13202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:52.004993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45798 (local address 127.2.110.195:42525)
0926 06:15:52.005061 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 06:15:52.005063 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:15:52.005071 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:15:52.005106 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:15:52.005107 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:15:52.005125 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:15:52.005202 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:15:52.005320 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:52.005926 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:52.006703 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:15:52.006905 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:15:52.006969 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:15:52.007123 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:15:52.007274 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:15:52.007349 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 06:15:52.007393 (+ 44us) server_negotiation.cc:299] Negotiation successful
0926 06:15:52.007415 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 06:15:52.009797 12872 raft_consensus.cc:1261] T 41beca9c5cef4e26876356bb647b9d91 P fe6063bdf7ce481085ee6831656d3935 [term 1 FOLLOWER]: Refusing update from remote peer 39a9eef5cfb6464ab50e5ac2de33de1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:15:52.009824 13006 raft_consensus.cc:1261] T 41beca9c5cef4e26876356bb647b9d91 P d5566ed2ede34420984e3fc58557454b [term 1 FOLLOWER]: Refusing update from remote peer 39a9eef5cfb6464ab50e5ac2de33de1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:15:52.010191 13213 consensus_queue.cc:1035] T 41beca9c5cef4e26876356bb647b9d91 P 39a9eef5cfb6464ab50e5ac2de33de1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe6063bdf7ce481085ee6831656d3935" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 34165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:15:52.010337 13210 consensus_queue.cc:1035] T 41beca9c5cef4e26876356bb647b9d91 P 39a9eef5cfb6464ab50e5ac2de33de1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5566ed2ede34420984e3fc58557454b" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 40521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:15:52.013226 13216 mvcc.cc:205] Tried to move back new op lower bound from 7333492129829466112 to 7333492129743151104. Current Snapshot: MvccSnapshot[applied={T|T < 7333492129829466112}]
I20260926 06:15:52.015863 12726 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:15:52.015985 12726 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:15:52.016880 12726 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f93460d1ed69447d84af432f2202f951: Sending DeleteTablet for 3 replicas of tablet 41beca9c5cef4e26876356bb647b9d91
I20260926 06:15:52.017194 12986 tablet_service.cc:1514] Processing DeleteTablet for tablet 41beca9c5cef4e26876356bb647b9d91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:15:52 UTC) from {username='kudu', principal='kudu/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:44130
I20260926 06:15:52.017201 13120 tablet_service.cc:1514] Processing DeleteTablet for tablet 41beca9c5cef4e26876356bb647b9d91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:15:52 UTC) from {username='kudu', principal='kudu/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:45796
I20260926 06:15:52.017297 12852 tablet_service.cc:1514] Processing DeleteTablet for tablet 41beca9c5cef4e26876356bb647b9d91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:15:52 UTC) from {username='kudu', principal='kudu/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:51104
I20260926 06:15:52.017452 13226 tablet_replica.cc:330] T 41beca9c5cef4e26876356bb647b9d91 P 39a9eef5cfb6464ab50e5ac2de33de1f: stopping tablet replica
I20260926 06:15:52.017575 13226 raft_consensus.cc:2229] T 41beca9c5cef4e26876356bb647b9d91 P 39a9eef5cfb6464ab50e5ac2de33de1f [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:15:52.017587 13227 tablet_replica.cc:330] T 41beca9c5cef4e26876356bb647b9d91 P fe6063bdf7ce481085ee6831656d3935: stopping tablet replica
I20260926 06:15:52.017681 13226 raft_consensus.cc:2258] T 41beca9c5cef4e26876356bb647b9d91 P 39a9eef5cfb6464ab50e5ac2de33de1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:15:52.017748 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 12789
I20260926 06:15:52.019826 13226 ts_tablet_manager.cc:1905] T 41beca9c5cef4e26876356bb647b9d91 P 39a9eef5cfb6464ab50e5ac2de33de1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:15:52.021989 13226 ts_tablet_manager.cc:1918] T 41beca9c5cef4e26876356bb647b9d91 P 39a9eef5cfb6464ab50e5ac2de33de1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:15:52.022082 13226 log.cc:1196] T 41beca9c5cef4e26876356bb647b9d91 P 39a9eef5cfb6464ab50e5ac2de33de1f: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403292369990-2491-0/minicluster-data/ts-2/wal/wals/41beca9c5cef4e26876356bb647b9d91
I20260926 06:15:52.022367 13226 ts_tablet_manager.cc:1939] T 41beca9c5cef4e26876356bb647b9d91 P 39a9eef5cfb6464ab50e5ac2de33de1f: Deleting consensus metadata
I20260926 06:15:52.022840 12711 catalog_manager.cc:4841] TS 39a9eef5cfb6464ab50e5ac2de33de1f (127.2.110.195:42525): tablet 41beca9c5cef4e26876356bb647b9d91 (table test-table [id=2a82035b29324831b3a31a5aa41157e0]) successfully deleted
W20260926 06:15:52.023991 12713 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.110.193:34165 (error 108)
I20260926 06:15:52.024582 13198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:52.024440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.193:34165 (local address 127.0.0.1:51110)
0926 06:15:52.024503 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:52.024548 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.193:34165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 06:15:52.024617 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 12923
W20260926 06:15:52.026643 12713 catalog_manager.cc:4584] TS fe6063bdf7ce481085ee6831656d3935 (127.2.110.193:34165): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 41beca9c5cef4e26876356bb647b9d91: Network error: Client connection negotiation failed: client connection to 127.2.110.193:34165: connect: Connection refused (error 111)
I20260926 06:15:52.032616 13198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:15:52.032437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.194:40521 (local address 127.0.0.1:44134)
0926 06:15:52.032529 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0926 06:15:52.032568 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.194:40521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260926 06:15:52.032634 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 13057
I20260926 06:15:52.037653 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 12695
2026-09-26T06:15:52Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2087 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:15:52 dist-test-slave-hndd krb5kdc[13232](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:15:52 dist-test-slave-hndd krb5kdc[13232](info): set up 2 sockets
Sep 26 06:15:52 dist-test-slave-hndd krb5kdc[13232](info): commencing operation
krb5kdc: starting...
W20260926 06:15:53.106657 2491 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Sep 26 06:15:53 dist-test-slave-hndd krb5kdc[13232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:15:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:15:53Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.110.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.110.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.110.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/krb5kdc/rangeradmin_127.2.110.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.110.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/krb5kdc/rangeradmin_127.2.110.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.110.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.110.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.110.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/krb5kdc/rangerlookup_127.2.110.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.110.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/krb5kdc/rangerlookup_127.2.110.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.110.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.110.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.110.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/krb5kdc/HTTP_127.2.110.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.110.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/krb5kdc/HTTP_127.2.110.212@KRBTEST.COM.keytab.
I20260926 06:15:53.285249 2491 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskRgcQD_/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/postgres -l logfile start
2026-09-26 06:15:56.027 UTC [13262] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 06:15:56.027 UTC [13262] LOG: listening on IPv4 address "127.2.110.212", port 33581
2026-09-26 06:15:56.032 UTC [13262] LOG: listening on Unix socket "/tmp/.s.PGSQL.33581"
2026-09-26 06:15:56.071 UTC [13265] LOG: database system was shut down at 2026-09-26 06:15:54 UTC
2026-09-26 06:15:56.075 UTC [13262] LOG: database system is ready to accept connections
I20260926 06:15:57.066784 2491 mini_postgres.cc:96] Postgres bound to 33581
2026-09-26 06:15:57.072 UTC [13273] FATAL: database "slave" does not exist
127.2.110.212:33581 - accepting connections
I20260926 06:15:57.073300 2491 mini_ranger.cc:161] Starting Ranger...
I20260926 06:15:57.089800 2491 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 06:15:57.639779 2491 mini_ranger.cc:87] Created ranger Postgres database
I20260926 06:15:57.639894 2491 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ranger-admin
2026-09-26 06:15:57,942 [I] DB FLAVOR :POSTGRES
2026-09-26 06:15:57,943 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 06:15:57,943 [I] Checking connection..
2026-09-26 06:15:57,943 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgcQD_/build/release/bin/postgresql.jar:/tmp/dist-test-taskRgcQD_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.110.212:33581/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 06:15:58,221 [I] Checking connection passed.
2026-09-26 06:15:58,222 [I] --------- Verifying version history table ---------
2026-09-26 06:15:58,222 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgcQD_/build/release/bin/postgresql.jar:/tmp/dist-test-taskRgcQD_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.110.212:33581/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 06:15:58,516 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 06:15:58,516 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgcQD_/build/release/bin/postgresql.jar:/tmp/dist-test-taskRgcQD_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.110.212:33581/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 06:15:58,791 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 06:15:58,791 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 06:15:58,792 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgcQD_/build/release/bin/postgresql.jar:/tmp/dist-test-taskRgcQD_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.110.212:33581/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRgcQD_/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 06:15:59.040 UTC [13337] WARNING: there is no transaction in progress
2026-09-26 06:15:59,047 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 06:15:59,047 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgcQD_/build/release/bin/postgresql.jar:/tmp/dist-test-taskRgcQD_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.110.212:33581/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 06:15:59,331 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 06:15:59,332 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 06:15:59,332 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgcQD_/build/release/bin/postgresql.jar:/tmp/dist-test-taskRgcQD_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.110.212:33581/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 06:15:59,598 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgcQD_/build/release/bin/postgresql.jar:/tmp/dist-test-taskRgcQD_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.110.212:33581/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 06:15:59,894 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgcQD_/build/release/bin/postgresql.jar:/tmp/dist-test-taskRgcQD_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.110.212:33581/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-hndd.c.gcp-upstream.internal','N') ;"
2026-09-26 06:16:00,186 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgcQD_/build/release/bin/postgresql.jar:/tmp/dist-test-taskRgcQD_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.110.212:33581/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 06:16:00,456 [I] Table x_portal_user does not exist in database ranger
2026-09-26 06:16:00,456 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgcQD_/build/release/bin/postgresql.jar:/tmp/dist-test-taskRgcQD_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.110.212:33581/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 06:16:00,734 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 06:16:00,734 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgcQD_/build/release/bin/postgresql.jar:/tmp/dist-test-taskRgcQD_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.110.212:33581/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 06:16:01,041 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgcQD_/build/release/bin/postgresql.jar:/tmp/dist-test-taskRgcQD_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.110.212:33581/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 06:16:01,323 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 06:16:01,323 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgcQD_/build/release/bin/postgresql.jar:/tmp/dist-test-taskRgcQD_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.110.212:33581/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRgcQD_/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 06:16:02.474 UTC [13445] WARNING: there is no transaction in progress
2026-09-26 06:16:02.491 UTC [13445] WARNING: there is no transaction in progress
2026-09-26 06:16:02.510 UTC [13445] WARNING: there is no transaction in progress
2026-09-26 06:16:02.529 UTC [13445] WARNING: there is no transaction in progress
2026-09-26 06:16:02.549 UTC [13445] WARNING: there is no transaction in progress
2026-09-26 06:16:02.572 UTC [13445] WARNING: there is no transaction in progress
2026-09-26 06:16:02.638 UTC [13445] WARNING: there is no transaction in progress
2026-09-26 06:16:02.647 UTC [13445] WARNING: there is no transaction in progress
2026-09-26 06:16:02.664 UTC [13445] WARNING: there is no transaction in progress
2026-09-26 06:16:02.676 UTC [13445] WARNING: there is no transaction in progress
2026-09-26 06:16:02.688 UTC [13445] WARNING: there is no transaction in progress
2026-09-26 06:16:02.704 UTC [13445] WARNING: there is no transaction in progress
2026-09-26 06:16:02.715 UTC [13445] WARNING: there is no transaction in progress
2026-09-26 06:16:02.723 UTC [13445] WARNING: there is no transaction in progress
2026-09-26 06:16:02.731 UTC [13445] WARNING: there is no transaction in progress
2026-09-26 06:16:02.739 UTC [13445] WARNING: there is no transaction in progress
2026-09-26 06:16:03,545 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 06:16:03,545 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgcQD_/build/release/bin/postgresql.jar:/tmp/dist-test-taskRgcQD_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.110.212:33581/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 06:16:03,808 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 06:16:03,809 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgcQD_/build/release/bin/postgresql.jar:/tmp/dist-test-taskRgcQD_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.110.212:33581/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 06:16:04,071 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 06:16:04,072 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgcQD_/build/release/bin/postgresql.jar:/tmp/dist-test-taskRgcQD_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.110.212:33581/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 06:16:04,349 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 06:16:04,350 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgcQD_/build/release/bin/postgresql.jar:/tmp/dist-test-taskRgcQD_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.110.212:33581/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 06:16:04,621 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgcQD_/build/release/bin/postgresql.jar:/tmp/dist-test-taskRgcQD_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.110.212:33581/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 06:16:04,887 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgcQD_/build/release/bin/postgresql.jar:/tmp/dist-test-taskRgcQD_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.110.212:33581/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-hndd.c.gcp-upstream.internal';"
2026-09-26 06:16:05,133 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 06:16:05,133 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgcQD_/build/release/bin/postgresql.jar:/tmp/dist-test-taskRgcQD_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.110.212:33581/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 06:16:05,413 [I] DB_PATCHES have already been applied
I20260926 06:16:05.420457 2491 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ranger-admin:/tmp/dist-test-taskRgcQD_/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskRgcQD_/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskRgcQD_/build/release/bin/postgresql.jar:/tmp/dist-test-taskRgcQD_/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 06:16:05.420536 2491 mini_ranger.cc:193] Using host: 127.2.110.212
I20260926 06:16:05.423774 2491 mini_ranger.cc:227] Ranger admin URL: http://127.2.110.212:44955
Sep 26, 2026 6:16:05 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 6:16:05 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 6:16:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 6:16:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 6:16:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 6:16:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 6:16:06 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 06:16:06.551955 2491 mini_ranger.cc:160] Time spent starting Ranger: real 9.479s user 0.000s sys 0.010s
Sep 26, 2026 6:16:27 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 6:16:28 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 6:16:28 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 6:16:28 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 6:16:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 6:16:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 6:16:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 6:16:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 6:16:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 6:16:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 6:16:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 6:16:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 6:16:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 6:16:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 6:16:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 6:16:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 6:16:28 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 6:16:29 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 06:16:29.780925 2491 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.110.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.110.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.110.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/krb5kdc/rangerkms_127.2.110.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.110.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/krb5kdc/rangerkms_127.2.110.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.110.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.110.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.110.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/krb5kdc/HTTP_127.2.110.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.110.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/krb5kdc/HTTP_127.2.110.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 06:16:29 dist-test-slave-hndd krb5kdc[13232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403389, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 06:16:29.889637 2491 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 06:16:29.901252 2491 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 06:16:30.999969 2491 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 06:16:31.000048 2491 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ranger-kms
I20260926 06:16:31.000062 2491 mini_ranger_kms.cc:227] Using postgres at 127.2.110.212:33581
2026-09-26 06:16:31,216 [I] DB FLAVOR :POSTGRES
2026-09-26 06:16:31,216 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 06:16:31,216 [I] Checking connection
2026-09-26 06:16:31,216 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgcQD_/build/release/bin/postgresql.jar:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.110.212:33581/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 06:16:31,476 [I] connection success
2026-09-26 06:16:31,476 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 06:16:31,476 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 06:16:31,476 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgcQD_/build/release/bin/postgresql.jar:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.110.212:33581/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 06:16:31,727 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 06:16:31,727 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 06:16:31,727 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 06:16:31,727 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgcQD_/build/release/bin/postgresql.jar:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.110.212:33581/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 06:16:32,015 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 06:16:32.526276 2491 mini_ranger_kms.cc:310] Created kms service
I20260926 06:16:32.719244 2491 mini_ranger_kms.cc:326] Created kudu user
I20260926 06:16:32.768852 2491 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 06:16:32 dist-test-slave-hndd krb5kdc[13232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403389, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.110.212@KRBTEST.COM
I20260926 06:16:33.242372 2491 mini_ranger_kms.cc:384] Added ranger policy
I20260926 06:16:33.242502 2491 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ranger-kms:/tmp/dist-test-taskRgcQD_/build/release/bin/postgresql.jar:/tmp/dist-test-taskRgcQD_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskRgcQD_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 06:16:33.242525 2491 mini_ranger_kms.cc:240] Using host: 127.2.110.212
I20260926 06:16:33.246228 2491 mini_ranger_kms.cc:276] Ranger KMS PID: 13626
I20260926 06:16:33.246381 2491 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.110.212:38503
Sep 26, 2026 6:16:34 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 6:16:34 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 6:16:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 6:16:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 6:16:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 6:16:34 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 06:16:35.575474 2491 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.686s user 0.008s sys 0.189s
I20260926 06:16:35.575598 2491 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 06:16:35.575672 2491 mini_ranger_kms.cc:401] 127.2.110.212:38503/kms/v1/keys
Sep 26 06:16:35 dist-test-slave-hndd krb5kdc[13232](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790403395, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.110.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 06:16:36 dist-test-slave-hndd krb5kdc[13232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403395, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.110.212@KRBTEST.COM for HTTP/127.2.110.212@KRBTEST.COM
Sep 26 06:16:36 dist-test-slave-hndd krb5kdc[13232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403395, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.110.212@KRBTEST.COM for HTTP/127.2.110.212@KRBTEST.COM
Sep 26 06:16:39 dist-test-slave-hndd krb5kdc[13232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
I20260926 06:16:39.177253 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:45023
--webserver_interface=127.2.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.110.254:45023
--ranger_config_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.110.212:38503/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:16:39.287508 13676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:16:39.287746 13676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:16:39.287771 13676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:16:39.289058 13676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:16:39.289113 13676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:16:39.289129 13676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:16:39.289141 13676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:16:39.289153 13676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:16:39.290912 13676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.110.212:38503/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.110.254:45023
--ranger_config_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:45023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:16:39.291504 13676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:16:39.291781 13676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:16:39.297879 13684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:16:39.297897 13681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:16:39.298012 13682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:16:39.298053 13676 server_base.cc:1034] running on GCE node
I20260926 06:16:39.298456 13676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:16:39.298825 13676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:16:39.300014 13676 hybrid_clock.cc:648] HybridClock initialized: now 1790403399299987 us; error 45 us; skew 500 ppm
Sep 26 06:16:39 dist-test-slave-hndd krb5kdc[13232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403399, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:16:39.303205 13676 init.cc:377] Logged in from keytab as kudu/127.2.110.254@KRBTEST.COM (short username kudu)
I20260926 06:16:39.303726 13676 webserver.cc:460] Webserver started at http://127.2.110.254:34969/ using document root <none> and password file <none>
I20260926 06:16:39.303972 13676 fs_manager.cc:359] Metadata directory not provided
I20260926 06:16:39.304031 13676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:16:39.304188 13676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 06:16:39 dist-test-slave-hndd krb5kdc[13232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403399, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.254@KRBTEST.COM for HTTP/127.2.110.212@KRBTEST.COM
I20260926 06:16:39.366343 13676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/master-0/data/instance:
uuid: "aec92f6a38b646a3843c07c440196b9a"
format_stamp: "Formatted at 2026-09-26 06:16:39 on dist-test-slave-hndd"
server_key: "8f2e30f598584bcd94a188523db4bdea"
server_key_iv: "7d65101fd95228c58ff026647dd5d23a"
server_key_version: "kuduclusterkey@0"
I20260926 06:16:39.366792 13676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/master-0/wal/instance:
uuid: "aec92f6a38b646a3843c07c440196b9a"
format_stamp: "Formatted at 2026-09-26 06:16:39 on dist-test-slave-hndd"
server_key: "8f2e30f598584bcd94a188523db4bdea"
server_key_iv: "7d65101fd95228c58ff026647dd5d23a"
server_key_version: "kuduclusterkey@0"
I20260926 06:16:39.369007 13676 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 06:16:39.385468 13691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:16:39.385689 13676 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 06:16:39.385802 13676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "aec92f6a38b646a3843c07c440196b9a"
format_stamp: "Formatted at 2026-09-26 06:16:39 on dist-test-slave-hndd"
server_key: "8f2e30f598584bcd94a188523db4bdea"
server_key_iv: "7d65101fd95228c58ff026647dd5d23a"
server_key_version: "kuduclusterkey@0"
I20260926 06:16:39.385902 13676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:16:39.416795 13676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:16:39.419202 13676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:16:39.419360 13676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:16:39.424288 13676 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.254:45023
I20260926 06:16:39.424360 13743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.254:45023 every 8 connection(s)
I20260926 06:16:39.424731 13676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:16:39.425096 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 13676
I20260926 06:16:39.425213 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/master-0/wal/instance
I20260926 06:16:39.425438 2491 external_mini_cluster.cc:1428] Setting key a5041adfb27261e7be8ba278179e97c0
I20260926 06:16:39.425586 13744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:16:39.428889 13744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aec92f6a38b646a3843c07c440196b9a: Bootstrap starting.
I20260926 06:16:39.429628 13744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aec92f6a38b646a3843c07c440196b9a: Neither blocks nor log segments found. Creating new log.
I20260926 06:16:39.429931 13744 log.cc:824] T 00000000000000000000000000000000 P aec92f6a38b646a3843c07c440196b9a: Log is configured to *not* fsync() on all Append() calls
I20260926 06:16:39.430675 13744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aec92f6a38b646a3843c07c440196b9a: No bootstrap required, opened a new log
I20260926 06:16:39.432816 13744 raft_consensus.cc:348] T 00000000000000000000000000000000 P aec92f6a38b646a3843c07c440196b9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aec92f6a38b646a3843c07c440196b9a" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 45023 } }
I20260926 06:16:39.433008 13744 raft_consensus.cc:374] T 00000000000000000000000000000000 P aec92f6a38b646a3843c07c440196b9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:16:39.433048 13744 raft_consensus.cc:729] T 00000000000000000000000000000000 P aec92f6a38b646a3843c07c440196b9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aec92f6a38b646a3843c07c440196b9a, State: Initialized, Role: FOLLOWER
I20260926 06:16:39.433207 13744 consensus_queue.cc:260] T 00000000000000000000000000000000 P aec92f6a38b646a3843c07c440196b9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aec92f6a38b646a3843c07c440196b9a" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 45023 } }
I20260926 06:16:39.433302 13744 raft_consensus.cc:388] T 00000000000000000000000000000000 P aec92f6a38b646a3843c07c440196b9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:16:39.433370 13744 raft_consensus.cc:482] T 00000000000000000000000000000000 P aec92f6a38b646a3843c07c440196b9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:16:39.433444 13744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aec92f6a38b646a3843c07c440196b9a [term 0 FOLLOWER]: Advancing to term 1
Sep 26 06:16:39 dist-test-slave-hndd krb5kdc[13232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:16:39.434075 13744 raft_consensus.cc:504] T 00000000000000000000000000000000 P aec92f6a38b646a3843c07c440196b9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aec92f6a38b646a3843c07c440196b9a" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 45023 } }
I20260926 06:16:39.434187 13744 leader_election.cc:302] T 00000000000000000000000000000000 P aec92f6a38b646a3843c07c440196b9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aec92f6a38b646a3843c07c440196b9a; no voters:
I20260926 06:16:39.434369 13744 leader_election.cc:288] T 00000000000000000000000000000000 P aec92f6a38b646a3843c07c440196b9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:16:39.434396 13749 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aec92f6a38b646a3843c07c440196b9a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:16:39.434700 13744 sys_catalog.cc:564] T 00000000000000000000000000000000 P aec92f6a38b646a3843c07c440196b9a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:16:39.434962 13749 raft_consensus.cc:686] T 00000000000000000000000000000000 P aec92f6a38b646a3843c07c440196b9a [term 1 LEADER]: Becoming Leader. State: Replica: aec92f6a38b646a3843c07c440196b9a, State: Running, Role: LEADER
I20260926 06:16:39.436290 13749 consensus_queue.cc:237] T 00000000000000000000000000000000 P aec92f6a38b646a3843c07c440196b9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aec92f6a38b646a3843c07c440196b9a" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 45023 } }
I20260926 06:16:39.436689 13750 sys_catalog.cc:455] T 00000000000000000000000000000000 P aec92f6a38b646a3843c07c440196b9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader aec92f6a38b646a3843c07c440196b9a. Latest consensus state: current_term: 1 leader_uuid: "aec92f6a38b646a3843c07c440196b9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aec92f6a38b646a3843c07c440196b9a" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 45023 } } }
I20260926 06:16:39.436769 13750 sys_catalog.cc:458] T 00000000000000000000000000000000 P aec92f6a38b646a3843c07c440196b9a [sys.catalog]: This master's current role is: LEADER
I20260926 06:16:39.436921 13744 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 06:16:39.437288 13751 sys_catalog.cc:455] T 00000000000000000000000000000000 P aec92f6a38b646a3843c07c440196b9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aec92f6a38b646a3843c07c440196b9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aec92f6a38b646a3843c07c440196b9a" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 45023 } } }
I20260926 06:16:39.437413 13751 sys_catalog.cc:458] T 00000000000000000000000000000000 P aec92f6a38b646a3843c07c440196b9a [sys.catalog]: This master's current role is: LEADER
I20260926 06:16:39.437987 13763 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:16:39.440029 13747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:16:39.425953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42102 (local address 127.2.110.254:45023)
0926 06:16:39.426149 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 06:16:39.426159 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 06:16:39.426192 (+ 33us) server_negotiation.cc:407] Connection header received
0926 06:16:39.426782 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:16:39.426791 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:16:39.426852 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:16:39.427012 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:16:39.428043 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:16:39.429440 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:16:39.430444 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:16:39.431054 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:16:39.431176 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:16:39.434024 (+ 2848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:16:39.434035 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:16:39.434035 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:16:39.434048 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:16:39.436072 (+ 2024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:16:39.436573 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:16:39.436576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:16:39.436577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:16:39.436676 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:16:39.436875 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:16:39.436876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:16:39.436876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:16:39.437184 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:16:39.437314 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 06:16:39.437850 (+ 536us) server_negotiation.cc:299] Negotiation successful
0926 06:16:39.437967 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":93,"threads_started":1}
I20260926 06:16:39.441169 13763 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:16:39.450610 13763 catalog_manager.cc:1347] Generated new cluster ID: e47df492d2cb490f939b4199f4186c20
I20260926 06:16:39.450682 13763 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:16:39.465827 13763 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:16:39.466785 13763 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:16:39.480171 13763 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aec92f6a38b646a3843c07c440196b9a: Generated new TSK 0
I20260926 06:16:39.480501 13763 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab.
Entry for principal kudu/127.2.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab.
I20260926 06:16:40.333205 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.110.212:38503/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.193:0
--local_ip_for_outbound_sockets=127.2.110.193
--webserver_interface=127.2.110.193
--webserver_port=0
--tserver_master_addrs=127.2.110.254:45023
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab
--principal=kudu/127.2.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:16:40.478268 13789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:16:40.478643 13789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:16:40.478688 13789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:16:40.481684 13789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:16:40.481772 13789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:16:40.481824 13789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.193
I20260926 06:16:40.484702 13789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.110.212:38503/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab
--principal=kudu/127.2.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.110.193
--webserver_port=0
--tserver_master_addrs=127.2.110.254:45023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.193
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:16:40.485050 13789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:16:40.485464 13789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:16:40.492238 13795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:16:40.492331 13794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:16:40.492359 13797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:16:40.492640 13789 server_base.cc:1034] running on GCE node
I20260926 06:16:40.492875 13789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:16:40.493218 13789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:16:40.494438 13789 hybrid_clock.cc:648] HybridClock initialized: now 1790403400494414 us; error 34 us; skew 500 ppm
Sep 26 06:16:40 dist-test-slave-hndd krb5kdc[13232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403400, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:16:40.498454 13789 init.cc:377] Logged in from keytab as kudu/127.2.110.193@KRBTEST.COM (short username kudu)
I20260926 06:16:40.499053 13789 webserver.cc:460] Webserver started at http://127.2.110.193:36979/ using document root <none> and password file <none>
I20260926 06:16:40.499342 13789 fs_manager.cc:359] Metadata directory not provided
I20260926 06:16:40.499416 13789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:16:40.499619 13789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 06:16:40 dist-test-slave-hndd krb5kdc[13232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403400, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.193@KRBTEST.COM for HTTP/127.2.110.212@KRBTEST.COM
I20260926 06:16:40.515285 13789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ts-0/data/instance:
uuid: "b79952721ba14aa0bf5211cc3ce70548"
format_stamp: "Formatted at 2026-09-26 06:16:40 on dist-test-slave-hndd"
server_key: "175883e3bdbebb52112596684b13c13b"
server_key_iv: "9328e095ee4c40aab8c61df67f0519dd"
server_key_version: "kuduclusterkey@0"
I20260926 06:16:40.515892 13789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ts-0/wal/instance:
uuid: "b79952721ba14aa0bf5211cc3ce70548"
format_stamp: "Formatted at 2026-09-26 06:16:40 on dist-test-slave-hndd"
server_key: "175883e3bdbebb52112596684b13c13b"
server_key_iv: "9328e095ee4c40aab8c61df67f0519dd"
server_key_version: "kuduclusterkey@0"
I20260926 06:16:40.517977 13789 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 06:16:40.537688 13804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:16:40.537982 13789 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260926 06:16:40.538142 13789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ts-0/wal
uuid: "b79952721ba14aa0bf5211cc3ce70548"
format_stamp: "Formatted at 2026-09-26 06:16:40 on dist-test-slave-hndd"
server_key: "175883e3bdbebb52112596684b13c13b"
server_key_iv: "9328e095ee4c40aab8c61df67f0519dd"
server_key_version: "kuduclusterkey@0"
I20260926 06:16:40.538337 13789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:16:40.554780 13789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:16:40.557612 13789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:16:40.557866 13789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:16:40.558167 13789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:16:40.558609 13789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:16:40.558708 13789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:16:40.558816 13789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:16:40.558899 13789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:16:40.567485 13789 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.193:33777
I20260926 06:16:40.568079 13789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
I20260926 06:16:40.568562 13919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.193:33777 every 8 connection(s)
I20260926 06:16:40.568817 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 13789
I20260926 06:16:40.568933 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ts-0/wal/instance
I20260926 06:16:40.569169 2491 external_mini_cluster.cc:1428] Setting key 3d72a9c9979491783b0fbc426139eb11
Sep 26 06:16:40 dist-test-slave-hndd krb5kdc[13232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403400, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.193@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:16:40.581676 13922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:16:40.569930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:36445 (local address 127.2.110.254:45023)
0926 06:16:40.570060 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 06:16:40.570063 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:16:40.570284 (+ 221us) server_negotiation.cc:407] Connection header received
0926 06:16:40.571010 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:16:40.571012 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:16:40.571027 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:16:40.571071 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:16:40.571979 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:16:40.572786 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:16:40.574629 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:16:40.574941 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:16:40.574995 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:16:40.577900 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:16:40.577901 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:16:40.577902 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:16:40.577912 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:16:40.579851 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:16:40.580407 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:16:40.580409 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:16:40.580409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:16:40.580474 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:16:40.580807 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:16:40.580808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:16:40.580809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:16:40.580963 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:16:40.581009 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 06:16:40.581462 (+ 453us) server_negotiation.cc:299] Negotiation successful
0926 06:16:40.581559 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
I20260926 06:16:40.582945 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:16:40.569915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:45023 (local address 127.2.110.193:36445)
0926 06:16:40.570147 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 06:16:40.570201 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 06:16:40.570884 (+ 683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:16:40.571272 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:16:40.571276 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:16:40.571547 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:16:40.571857 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:16:40.571864 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:16:40.572915 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:16:40.572916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:16:40.574497 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:16:40.574502 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:16:40.575106 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:16:40.575107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:16:40.575252 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:16:40.575930 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:16:40.575941 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:16:40.577771 (+ 1830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:16:40.579962 (+ 2191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:16:40.579970 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:16:40.579972 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:16:40.580311 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:16:40.580547 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:16:40.580548 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:16:40.580548 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:16:40.580718 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:16:40.581062 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:16:40.581070 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:16:40.581258 (+ 188us) client_negotiation.cc:769] Sending connection context
0926 06:16:40.582096 (+ 838us) client_negotiation.cc:240] Negotiation successful
0926 06:16:40.582685 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":97,"threads_started":1}
I20260926 06:16:40.585530 13920 heartbeater.cc:347] Connected to a master server at 127.2.110.254:45023
I20260926 06:16:40.585659 13920 heartbeater.cc:464] Registering TS with master...
I20260926 06:16:40.585873 13920 heartbeater.cc:511] Master 127.2.110.254:45023 requested a full tablet report, sending...
I20260926 06:16:40.586519 13708 ts_manager.cc:194] Registered new tserver with Master: b79952721ba14aa0bf5211cc3ce70548 (127.2.110.193:33777)
I20260926 06:16:40.587430 13708 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.193@KRBTEST.COM'} at 127.2.110.193:36445
WARNING: no policy specified for kudu/127.2.110.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab.
Entry for principal kudu/127.2.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab.
I20260926 06:16:40.620932 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.110.212:38503/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.194:0
--local_ip_for_outbound_sockets=127.2.110.194
--webserver_interface=127.2.110.194
--webserver_port=0
--tserver_master_addrs=127.2.110.254:45023
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab
--principal=kudu/127.2.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 06:16:40 dist-test-slave-hndd krb5kdc[13232](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790403400, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 06:16:40.754463 13926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:16:40.754814 13926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:16:40.754860 13926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:16:40.756963 13926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:16:40.757105 13926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:16:40.757193 13926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.194
I20260926 06:16:40.759961 13926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.110.212:38503/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab
--principal=kudu/127.2.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.110.194
--webserver_port=0
--tserver_master_addrs=127.2.110.254:45023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.194
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:16:40.760425 13926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:16:40.760764 13926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:16:40.767777 13934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:16:40.767889 13933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:16:40.767784 13936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:16:40.769161 13926 server_base.cc:1034] running on GCE node
I20260926 06:16:40.769454 13926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:16:40.769783 13926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:16:40.770974 13926 hybrid_clock.cc:648] HybridClock initialized: now 1790403400770950 us; error 35 us; skew 500 ppm
Sep 26 06:16:40 dist-test-slave-hndd krb5kdc[13232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403400, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:16:40.774381 13926 init.cc:377] Logged in from keytab as kudu/127.2.110.194@KRBTEST.COM (short username kudu)
I20260926 06:16:40.775002 13926 webserver.cc:460] Webserver started at http://127.2.110.194:35159/ using document root <none> and password file <none>
I20260926 06:16:40.775251 13926 fs_manager.cc:359] Metadata directory not provided
I20260926 06:16:40.775308 13926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:16:40.775490 13926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 06:16:40 dist-test-slave-hndd krb5kdc[13232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403400, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.194@KRBTEST.COM for HTTP/127.2.110.212@KRBTEST.COM
I20260926 06:16:40.789808 13926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ts-1/data/instance:
uuid: "63e3996d9f3d4b549e96bbb2d8a24f16"
format_stamp: "Formatted at 2026-09-26 06:16:40 on dist-test-slave-hndd"
server_key: "a1cf804931b3a3d9796a467e457b745c"
server_key_iv: "9e34da3380541badb60c9c5f4e003769"
server_key_version: "kuduclusterkey@0"
I20260926 06:16:40.790527 13926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ts-1/wal/instance:
uuid: "63e3996d9f3d4b549e96bbb2d8a24f16"
format_stamp: "Formatted at 2026-09-26 06:16:40 on dist-test-slave-hndd"
server_key: "a1cf804931b3a3d9796a467e457b745c"
server_key_iv: "9e34da3380541badb60c9c5f4e003769"
server_key_version: "kuduclusterkey@0"
I20260926 06:16:40.794070 13926 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260926 06:16:40.812713 13944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:16:40.813639 13926 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 06:16:40.813748 13926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ts-1/wal
uuid: "63e3996d9f3d4b549e96bbb2d8a24f16"
format_stamp: "Formatted at 2026-09-26 06:16:40 on dist-test-slave-hndd"
server_key: "a1cf804931b3a3d9796a467e457b745c"
server_key_iv: "9e34da3380541badb60c9c5f4e003769"
server_key_version: "kuduclusterkey@0"
I20260926 06:16:40.813862 13926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:16:40.831156 13926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:16:40.833895 13926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:16:40.834151 13926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:16:40.834468 13926 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:16:40.834972 13926 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:16:40.835083 13926 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:16:40.835166 13926 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:16:40.835233 13926 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:16:40.843770 13926 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.194:40773
I20260926 06:16:40.844417 13926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
I20260926 06:16:40.845218 14057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.194:40773 every 8 connection(s)
I20260926 06:16:40.849417 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 13926
I20260926 06:16:40.849545 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ts-1/wal/instance
I20260926 06:16:40.849763 2491 external_mini_cluster.cc:1428] Setting key 8be5aa631b9989f353406c546f515e76
Sep 26 06:16:40 dist-test-slave-hndd krb5kdc[13232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403400, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.194@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:16:40.857440 13922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:16:40.846200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:35257 (local address 127.2.110.254:45023)
0926 06:16:40.846248 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 06:16:40.846250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:16:40.846451 (+ 201us) server_negotiation.cc:407] Connection header received
0926 06:16:40.847180 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:16:40.847182 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:16:40.847198 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:16:40.847230 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:16:40.848129 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:16:40.848905 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:16:40.850879 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:16:40.851167 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:16:40.851211 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:16:40.854019 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:16:40.854021 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:16:40.854021 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:16:40.854031 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:16:40.855755 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:16:40.856299 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:16:40.856300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:16:40.856301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:16:40.856364 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:16:40.856660 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:16:40.856661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:16:40.856662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:16:40.856798 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:16:40.856883 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 06:16:40.857198 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 06:16:40.857299 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 06:16:40.857661 14060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:16:40.845912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:45023 (local address 127.2.110.194:35257)
0926 06:16:40.846314 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0926 06:16:40.846363 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 06:16:40.847043 (+ 680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:16:40.847429 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:16:40.847434 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:16:40.847715 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:16:40.848010 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:16:40.848017 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:16:40.849030 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:16:40.849031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:16:40.850774 (+ 1743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:16:40.850778 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:16:40.851305 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:16:40.851306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:16:40.851460 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:16:40.852106 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:16:40.852117 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:16:40.853893 (+ 1776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:16:40.855902 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:16:40.855912 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:16:40.855915 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:16:40.856192 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:16:40.856441 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:16:40.856442 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:16:40.856442 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:16:40.856581 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:16:40.856873 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:16:40.856880 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:16:40.857051 (+ 171us) client_negotiation.cc:769] Sending connection context
0926 06:16:40.857178 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 06:16:40.857382 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":90,"threads_started":1}
I20260926 06:16:40.858162 14058 heartbeater.cc:347] Connected to a master server at 127.2.110.254:45023
I20260926 06:16:40.858297 14058 heartbeater.cc:464] Registering TS with master...
I20260926 06:16:40.858505 14058 heartbeater.cc:511] Master 127.2.110.254:45023 requested a full tablet report, sending...
I20260926 06:16:40.858955 13708 ts_manager.cc:194] Registered new tserver with Master: 63e3996d9f3d4b549e96bbb2d8a24f16 (127.2.110.194:40773)
I20260926 06:16:40.859377 13708 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.194@KRBTEST.COM'} at 127.2.110.194:35257
WARNING: no policy specified for kudu/127.2.110.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab.
Entry for principal kudu/127.2.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab.
I20260926 06:16:40.893838 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.110.212:38503/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.195:0
--local_ip_for_outbound_sockets=127.2.110.195
--webserver_interface=127.2.110.195
--webserver_port=0
--tserver_master_addrs=127.2.110.254:45023
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab
--principal=kudu/127.2.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:16:40.994283 14063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:16:40.994678 14063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:16:40.994771 14063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:16:40.996682 14063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:16:40.996778 14063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:16:40.996827 14063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.195
I20260926 06:16:40.998663 14063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.110.212:38503/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab
--principal=kudu/127.2.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.110.195
--webserver_port=0
--tserver_master_addrs=127.2.110.254:45023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.195
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:16:40.998979 14063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:16:40.999236 14063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:16:41.006040 14071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:16:41.006099 14063 server_base.cc:1034] running on GCE node
W20260926 06:16:41.006263 14068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:16:41.006269 14069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:16:41.006714 14063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:16:41.007054 14063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:16:41.008255 14063 hybrid_clock.cc:648] HybridClock initialized: now 1790403401008230 us; error 41 us; skew 500 ppm
Sep 26 06:16:41 dist-test-slave-hndd krb5kdc[13232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403401, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:16:41.012006 14063 init.cc:377] Logged in from keytab as kudu/127.2.110.195@KRBTEST.COM (short username kudu)
I20260926 06:16:41.012544 14063 webserver.cc:460] Webserver started at http://127.2.110.195:43581/ using document root <none> and password file <none>
I20260926 06:16:41.012825 14063 fs_manager.cc:359] Metadata directory not provided
I20260926 06:16:41.012878 14063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:16:41.013015 14063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 06:16:41 dist-test-slave-hndd krb5kdc[13232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403401, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.195@KRBTEST.COM for HTTP/127.2.110.212@KRBTEST.COM
I20260926 06:16:41.027634 14063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ts-2/data/instance:
uuid: "136c8b33bab647ab9bc8dce56ec46d12"
format_stamp: "Formatted at 2026-09-26 06:16:41 on dist-test-slave-hndd"
server_key: "7530b6239dd35943543ac5788cef67ee"
server_key_iv: "890a0745b3591a7685babc6632324be3"
server_key_version: "kuduclusterkey@0"
I20260926 06:16:41.028285 14063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ts-2/wal/instance:
uuid: "136c8b33bab647ab9bc8dce56ec46d12"
format_stamp: "Formatted at 2026-09-26 06:16:41 on dist-test-slave-hndd"
server_key: "7530b6239dd35943543ac5788cef67ee"
server_key_iv: "890a0745b3591a7685babc6632324be3"
server_key_version: "kuduclusterkey@0"
I20260926 06:16:41.039949 14063 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.002s sys 0.000s
I20260926 06:16:41.056366 14078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:16:41.056624 14063 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:16:41.056720 14063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ts-2/wal
uuid: "136c8b33bab647ab9bc8dce56ec46d12"
format_stamp: "Formatted at 2026-09-26 06:16:41 on dist-test-slave-hndd"
server_key: "7530b6239dd35943543ac5788cef67ee"
server_key_iv: "890a0745b3591a7685babc6632324be3"
server_key_version: "kuduclusterkey@0"
I20260926 06:16:41.056813 14063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:16:41.087797 14063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:16:41.095219 14063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:16:41.095481 14063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:16:41.095886 14063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:16:41.096308 14063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:16:41.096398 14063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:16:41.096487 14063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:16:41.096525 14063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:16:41.103849 14063 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.195:36177
I20260926 06:16:41.104292 14063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
I20260926 06:16:41.105890 14191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.195:36177 every 8 connection(s)
I20260926 06:16:41.107324 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 14063
I20260926 06:16:41.107456 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403292369990-2491-0/minicluster-data/ts-2/wal/instance
I20260926 06:16:41.107690 2491 external_mini_cluster.cc:1428] Setting key 5f1a9c09b7f973697e10ef52a6c54dc4
Sep 26 06:16:41 dist-test-slave-hndd krb5kdc[13232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403401, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.195@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:16:41.117844 13922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:16:41.105128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:44979 (local address 127.2.110.254:45023)
0926 06:16:41.105186 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 06:16:41.105189 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:16:41.106356 (+ 1167us) server_negotiation.cc:407] Connection header received
0926 06:16:41.107163 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:16:41.107165 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:16:41.107177 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:16:41.107211 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:16:41.108137 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:16:41.108953 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:16:41.110653 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:16:41.110994 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:16:41.111044 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:16:41.114038 (+ 2994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:16:41.114039 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:16:41.114039 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:16:41.114048 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:16:41.115904 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:16:41.116522 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:16:41.116523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:16:41.116523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:16:41.116591 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:16:41.116947 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:16:41.116948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:16:41.116948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:16:41.117114 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:16:41.117191 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 06:16:41.117647 (+ 456us) server_negotiation.cc:299] Negotiation successful
0926 06:16:41.117746 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 06:16:41.118049 14194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:16:41.105121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:45023 (local address 127.2.110.195:44979)
0926 06:16:41.106206 (+ 1085us) negotiation.cc:107] Waiting for socket to connect
0926 06:16:41.106257 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 06:16:41.107024 (+ 767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:16:41.107407 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:16:41.107411 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:16:41.107690 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:16:41.107983 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:16:41.107989 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:16:41.109081 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:16:41.109082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:16:41.110532 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:16:41.110535 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:16:41.111153 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:16:41.111154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:16:41.111316 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:16:41.112050 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:16:41.112062 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:16:41.113910 (+ 1848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:16:41.116016 (+ 2106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:16:41.116026 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:16:41.116029 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:16:41.116406 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:16:41.116676 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:16:41.116677 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:16:41.116677 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:16:41.116842 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:16:41.117204 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:16:41.117212 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:16:41.117447 (+ 235us) client_negotiation.cc:769] Sending connection context
0926 06:16:41.117586 (+ 139us) client_negotiation.cc:240] Negotiation successful
0926 06:16:41.117776 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1039,"thread_start_us":103,"threads_started":1}
I20260926 06:16:41.118741 14192 heartbeater.cc:347] Connected to a master server at 127.2.110.254:45023
I20260926 06:16:41.118883 14192 heartbeater.cc:464] Registering TS with master...
I20260926 06:16:41.119120 14192 heartbeater.cc:511] Master 127.2.110.254:45023 requested a full tablet report, sending...
I20260926 06:16:41.119582 13708 ts_manager.cc:194] Registered new tserver with Master: 136c8b33bab647ab9bc8dce56ec46d12 (127.2.110.195:36177)
I20260926 06:16:41.120118 13708 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.195@KRBTEST.COM'} at 127.2.110.195:44979
I20260926 06:16:41.130406 2491 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 06:16:41.141001 13922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:16:41.132305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42108 (local address 127.2.110.254:45023)
0926 06:16:41.132357 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 06:16:41.132359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:16:41.132568 (+ 209us) server_negotiation.cc:407] Connection header received
0926 06:16:41.132623 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:16:41.132624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:16:41.132636 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:16:41.132701 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:16:41.133544 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:16:41.134326 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:16:41.135496 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:16:41.135812 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:16:41.135887 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:16:41.137775 (+ 1888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:16:41.137777 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:16:41.137777 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:16:41.137785 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:16:41.139565 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:16:41.140004 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:16:41.140005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:16:41.140005 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:16:41.140080 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:16:41.140336 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:16:41.140337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:16:41.140337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:16:41.140510 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:16:41.140628 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 06:16:41.140794 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 06:16:41.140892 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 06:16:41.142935 13708 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42108:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:16:41.145577 13708 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:16:41.155467 14206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:16:41.149469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.194:40773 (local address 127.0.0.1:55202)
0926 06:16:41.150436 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0926 06:16:41.150479 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 06:16:41.150546 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:16:41.150882 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:16:41.150886 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:16:41.150889 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:16:41.150967 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:16:41.150970 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:16:41.152445 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:16:41.152446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:16:41.153910 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:16:41.153913 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:16:41.155096 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:16:41.155097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:16:41.155185 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:16:41.155201 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 06:16:41.155282 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 06:16:41.155352 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":936,"mutex_wait_us":6,"thread_start_us":84,"threads_started":1}
I20260926 06:16:41.155977 14208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:16:41.150088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55202 (local address 127.2.110.194:40773)
0926 06:16:41.150691 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0926 06:16:41.150695 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:16:41.150705 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:16:41.150744 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:16:41.150746 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:16:41.150762 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:16:41.150842 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:16:41.151078 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:16:41.152333 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:16:41.154046 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:16:41.154887 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:16:41.154969 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:16:41.155055 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 06:16:41.155836 (+ 781us) server_negotiation.cc:299] Negotiation successful
0926 06:16:41.155887 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":80,"threads_started":1}
I20260926 06:16:41.156939 13992 tablet_service.cc:1467] Processing CreateTablet for tablet dee70ca81ce04524a880e997f2592e68 (DEFAULT_TABLE table=test-table [id=89220cd564ff427abd31ccaadbe94380]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:16:41.157420 13992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dee70ca81ce04524a880e997f2592e68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:16:41.159001 14205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:16:41.149720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.193:33777 (local address 127.0.0.1:54868)
0926 06:16:41.151387 (+ 1667us) negotiation.cc:107] Waiting for socket to connect
0926 06:16:41.151397 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:16:41.151442 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:16:41.151673 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:16:41.151674 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:16:41.151676 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:16:41.151721 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:16:41.151723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:16:41.154631 (+ 2908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:16:41.154632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:16:41.157554 (+ 2922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:16:41.157557 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:16:41.158719 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:16:41.158721 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:16:41.158792 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:16:41.158798 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 06:16:41.158864 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 06:16:41.158916 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1651,"thread_start_us":90,"threads_started":1}
I20260926 06:16:41.159417 14209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:16:41.150254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54868 (local address 127.2.110.193:33777)
0926 06:16:41.150964 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0926 06:16:41.150968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:16:41.151497 (+ 529us) server_negotiation.cc:407] Connection header received
0926 06:16:41.151532 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:16:41.151534 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:16:41.151548 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:16:41.151614 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:16:41.151794 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:16:41.153238 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:16:41.157660 (+ 4422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:16:41.158624 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:16:41.158702 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:16:41.158759 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 06:16:41.159282 (+ 523us) server_negotiation.cc:299] Negotiation successful
0926 06:16:41.159345 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":682,"thread_start_us":54,"threads_started":1}
I20260926 06:16:41.160463 14204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:16:41.149487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.195:36177 (local address 127.0.0.1:39066)
0926 06:16:41.151016 (+ 1529us) negotiation.cc:107] Waiting for socket to connect
0926 06:16:41.151026 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:16:41.151094 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:16:41.153359 (+ 2265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:16:41.153360 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:16:41.153362 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:16:41.153420 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:16:41.153422 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:16:41.156083 (+ 2661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:16:41.156084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:16:41.158926 (+ 2842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:16:41.158929 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:16:41.160173 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:16:41.160174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:16:41.160253 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:16:41.160258 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 06:16:41.160334 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 06:16:41.160380 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1513,"thread_start_us":98,"threads_started":1}
I20260926 06:16:41.160473 14207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:16:41.149751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39066 (local address 127.2.110.195:36177)
0926 06:16:41.149978 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 06:16:41.149983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:16:41.151154 (+ 1171us) server_negotiation.cc:407] Connection header received
0926 06:16:41.151200 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:16:41.151203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:16:41.151219 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:16:41.151287 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:16:41.153504 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:16:41.155969 (+ 2465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:16:41.159042 (+ 3073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:16:41.160080 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:16:41.160158 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:16:41.160228 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 06:16:41.160348 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 06:16:41.160389 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":81,"threads_started":1}
I20260926 06:16:41.160444 13842 tablet_service.cc:1467] Processing CreateTablet for tablet dee70ca81ce04524a880e997f2592e68 (DEFAULT_TABLE table=test-table [id=89220cd564ff427abd31ccaadbe94380]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:16:41.160808 13842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dee70ca81ce04524a880e997f2592e68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:16:41.162192 14125 tablet_service.cc:1467] Processing CreateTablet for tablet dee70ca81ce04524a880e997f2592e68 (DEFAULT_TABLE table=test-table [id=89220cd564ff427abd31ccaadbe94380]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:16:41.162552 14125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dee70ca81ce04524a880e997f2592e68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:16:41.164311 14210 tablet_bootstrap.cc:492] T dee70ca81ce04524a880e997f2592e68 P b79952721ba14aa0bf5211cc3ce70548: Bootstrap starting.
I20260926 06:16:41.165279 14210 tablet_bootstrap.cc:654] T dee70ca81ce04524a880e997f2592e68 P b79952721ba14aa0bf5211cc3ce70548: Neither blocks nor log segments found. Creating new log.
I20260926 06:16:41.165692 14210 log.cc:824] T dee70ca81ce04524a880e997f2592e68 P b79952721ba14aa0bf5211cc3ce70548: Log is configured to *not* fsync() on all Append() calls
I20260926 06:16:41.166234 14211 tablet_bootstrap.cc:492] T dee70ca81ce04524a880e997f2592e68 P 63e3996d9f3d4b549e96bbb2d8a24f16: Bootstrap starting.
I20260926 06:16:41.167204 14211 tablet_bootstrap.cc:654] T dee70ca81ce04524a880e997f2592e68 P 63e3996d9f3d4b549e96bbb2d8a24f16: Neither blocks nor log segments found. Creating new log.
I20260926 06:16:41.167497 14211 log.cc:824] T dee70ca81ce04524a880e997f2592e68 P 63e3996d9f3d4b549e96bbb2d8a24f16: Log is configured to *not* fsync() on all Append() calls
I20260926 06:16:41.168282 14210 tablet_bootstrap.cc:492] T dee70ca81ce04524a880e997f2592e68 P b79952721ba14aa0bf5211cc3ce70548: No bootstrap required, opened a new log
I20260926 06:16:41.168365 14210 ts_tablet_manager.cc:1397] T dee70ca81ce04524a880e997f2592e68 P b79952721ba14aa0bf5211cc3ce70548: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260926 06:16:41.170912 14210 raft_consensus.cc:348] T dee70ca81ce04524a880e997f2592e68 P b79952721ba14aa0bf5211cc3ce70548 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63e3996d9f3d4b549e96bbb2d8a24f16" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 40773 } } peers { permanent_uuid: "b79952721ba14aa0bf5211cc3ce70548" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 33777 } } peers { permanent_uuid: "136c8b33bab647ab9bc8dce56ec46d12" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36177 } }
I20260926 06:16:41.171123 14210 raft_consensus.cc:374] T dee70ca81ce04524a880e997f2592e68 P b79952721ba14aa0bf5211cc3ce70548 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:16:41.171157 14210 raft_consensus.cc:729] T dee70ca81ce04524a880e997f2592e68 P b79952721ba14aa0bf5211cc3ce70548 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b79952721ba14aa0bf5211cc3ce70548, State: Initialized, Role: FOLLOWER
I20260926 06:16:41.171306 14210 consensus_queue.cc:260] T dee70ca81ce04524a880e997f2592e68 P b79952721ba14aa0bf5211cc3ce70548 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63e3996d9f3d4b549e96bbb2d8a24f16" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 40773 } } peers { permanent_uuid: "b79952721ba14aa0bf5211cc3ce70548" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 33777 } } peers { permanent_uuid: "136c8b33bab647ab9bc8dce56ec46d12" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36177 } }
I20260926 06:16:41.171654 13920 heartbeater.cc:503] Master 127.2.110.254:45023 was elected leader, sending a full tablet report...
I20260926 06:16:41.171975 14210 ts_tablet_manager.cc:1428] T dee70ca81ce04524a880e997f2592e68 P b79952721ba14aa0bf5211cc3ce70548: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260926 06:16:41.173538 14213 tablet_bootstrap.cc:492] T dee70ca81ce04524a880e997f2592e68 P 136c8b33bab647ab9bc8dce56ec46d12: Bootstrap starting.
I20260926 06:16:41.174180 14211 tablet_bootstrap.cc:492] T dee70ca81ce04524a880e997f2592e68 P 63e3996d9f3d4b549e96bbb2d8a24f16: No bootstrap required, opened a new log
I20260926 06:16:41.174340 14211 ts_tablet_manager.cc:1397] T dee70ca81ce04524a880e997f2592e68 P 63e3996d9f3d4b549e96bbb2d8a24f16: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260926 06:16:41.174655 14213 tablet_bootstrap.cc:654] T dee70ca81ce04524a880e997f2592e68 P 136c8b33bab647ab9bc8dce56ec46d12: Neither blocks nor log segments found. Creating new log.
I20260926 06:16:41.174993 14213 log.cc:824] T dee70ca81ce04524a880e997f2592e68 P 136c8b33bab647ab9bc8dce56ec46d12: Log is configured to *not* fsync() on all Append() calls
I20260926 06:16:41.176937 14213 tablet_bootstrap.cc:492] T dee70ca81ce04524a880e997f2592e68 P 136c8b33bab647ab9bc8dce56ec46d12: No bootstrap required, opened a new log
I20260926 06:16:41.177047 14213 ts_tablet_manager.cc:1397] T dee70ca81ce04524a880e997f2592e68 P 136c8b33bab647ab9bc8dce56ec46d12: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 06:16:41.176944 14211 raft_consensus.cc:348] T dee70ca81ce04524a880e997f2592e68 P 63e3996d9f3d4b549e96bbb2d8a24f16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63e3996d9f3d4b549e96bbb2d8a24f16" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 40773 } } peers { permanent_uuid: "b79952721ba14aa0bf5211cc3ce70548" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 33777 } } peers { permanent_uuid: "136c8b33bab647ab9bc8dce56ec46d12" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36177 } }
I20260926 06:16:41.177652 14211 raft_consensus.cc:374] T dee70ca81ce04524a880e997f2592e68 P 63e3996d9f3d4b549e96bbb2d8a24f16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:16:41.177721 14211 raft_consensus.cc:729] T dee70ca81ce04524a880e997f2592e68 P 63e3996d9f3d4b549e96bbb2d8a24f16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63e3996d9f3d4b549e96bbb2d8a24f16, State: Initialized, Role: FOLLOWER
I20260926 06:16:41.177894 14211 consensus_queue.cc:260] T dee70ca81ce04524a880e997f2592e68 P 63e3996d9f3d4b549e96bbb2d8a24f16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63e3996d9f3d4b549e96bbb2d8a24f16" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 40773 } } peers { permanent_uuid: "b79952721ba14aa0bf5211cc3ce70548" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 33777 } } peers { permanent_uuid: "136c8b33bab647ab9bc8dce56ec46d12" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36177 } }
I20260926 06:16:41.178220 14211 ts_tablet_manager.cc:1428] T dee70ca81ce04524a880e997f2592e68 P 63e3996d9f3d4b549e96bbb2d8a24f16: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260926 06:16:41.178362 14058 heartbeater.cc:503] Master 127.2.110.254:45023 was elected leader, sending a full tablet report...
I20260926 06:16:41.179690 14213 raft_consensus.cc:348] T dee70ca81ce04524a880e997f2592e68 P 136c8b33bab647ab9bc8dce56ec46d12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63e3996d9f3d4b549e96bbb2d8a24f16" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 40773 } } peers { permanent_uuid: "b79952721ba14aa0bf5211cc3ce70548" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 33777 } } peers { permanent_uuid: "136c8b33bab647ab9bc8dce56ec46d12" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36177 } }
I20260926 06:16:41.179867 14213 raft_consensus.cc:374] T dee70ca81ce04524a880e997f2592e68 P 136c8b33bab647ab9bc8dce56ec46d12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:16:41.179901 14213 raft_consensus.cc:729] T dee70ca81ce04524a880e997f2592e68 P 136c8b33bab647ab9bc8dce56ec46d12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 136c8b33bab647ab9bc8dce56ec46d12, State: Initialized, Role: FOLLOWER
I20260926 06:16:41.180048 14213 consensus_queue.cc:260] T dee70ca81ce04524a880e997f2592e68 P 136c8b33bab647ab9bc8dce56ec46d12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63e3996d9f3d4b549e96bbb2d8a24f16" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 40773 } } peers { permanent_uuid: "b79952721ba14aa0bf5211cc3ce70548" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 33777 } } peers { permanent_uuid: "136c8b33bab647ab9bc8dce56ec46d12" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36177 } }
I20260926 06:16:41.180327 14213 ts_tablet_manager.cc:1428] T dee70ca81ce04524a880e997f2592e68 P 136c8b33bab647ab9bc8dce56ec46d12: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 06:16:41.180377 14192 heartbeater.cc:503] Master 127.2.110.254:45023 was elected leader, sending a full tablet report...
I20260926 06:16:41.243835 14218 raft_consensus.cc:482] T dee70ca81ce04524a880e997f2592e68 P 136c8b33bab647ab9bc8dce56ec46d12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:16:41.243966 14218 raft_consensus.cc:504] T dee70ca81ce04524a880e997f2592e68 P 136c8b33bab647ab9bc8dce56ec46d12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63e3996d9f3d4b549e96bbb2d8a24f16" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 40773 } } peers { permanent_uuid: "b79952721ba14aa0bf5211cc3ce70548" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 33777 } } peers { permanent_uuid: "136c8b33bab647ab9bc8dce56ec46d12" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36177 } }
I20260926 06:16:41.244387 14218 leader_election.cc:288] T dee70ca81ce04524a880e997f2592e68 P 136c8b33bab647ab9bc8dce56ec46d12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63e3996d9f3d4b549e96bbb2d8a24f16 (127.2.110.194:40773), b79952721ba14aa0bf5211cc3ce70548 (127.2.110.193:33777)
I20260926 06:16:41.248840 14194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:16:41.244686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.194:40773 (local address 127.2.110.195:60397)
0926 06:16:41.245011 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0926 06:16:41.245025 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 06:16:41.245075 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:16:41.245604 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:16:41.245605 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:16:41.245607 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:16:41.245680 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:16:41.245682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:16:41.246629 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:16:41.246630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:16:41.247789 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:16:41.247792 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:16:41.248566 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:16:41.248567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:16:41.248656 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:16:41.248659 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:16:41.248711 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 06:16:41.248757 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305}
I20260926 06:16:41.249480 14208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:16:41.245253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:60397 (local address 127.2.110.194:40773)
0926 06:16:41.245297 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 06:16:41.245299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:16:41.245308 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:16:41.245359 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:16:41.245360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:16:41.245371 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:16:41.245425 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:16:41.245767 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:16:41.246512 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:16:41.247895 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:16:41.248433 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:16:41.248522 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:16:41.248532 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 06:16:41.249364 (+ 832us) server_negotiation.cc:299] Negotiation successful
0926 06:16:41.249410 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 06:16:41.249809 14007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee70ca81ce04524a880e997f2592e68" candidate_uuid: "136c8b33bab647ab9bc8dce56ec46d12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63e3996d9f3d4b549e96bbb2d8a24f16" is_pre_election: true
I20260926 06:16:41.249990 14007 raft_consensus.cc:2454] T dee70ca81ce04524a880e997f2592e68 P 63e3996d9f3d4b549e96bbb2d8a24f16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 136c8b33bab647ab9bc8dce56ec46d12 in term 0.
I20260926 06:16:41.250281 14081 leader_election.cc:302] T dee70ca81ce04524a880e997f2592e68 P 136c8b33bab647ab9bc8dce56ec46d12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 136c8b33bab647ab9bc8dce56ec46d12, 63e3996d9f3d4b549e96bbb2d8a24f16; no voters:
I20260926 06:16:41.250411 14218 raft_consensus.cc:2781] T dee70ca81ce04524a880e997f2592e68 P 136c8b33bab647ab9bc8dce56ec46d12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:16:41.250449 14218 raft_consensus.cc:482] T dee70ca81ce04524a880e997f2592e68 P 136c8b33bab647ab9bc8dce56ec46d12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:16:41.250469 14218 raft_consensus.cc:3037] T dee70ca81ce04524a880e997f2592e68 P 136c8b33bab647ab9bc8dce56ec46d12 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:16:41.251422 14218 raft_consensus.cc:504] T dee70ca81ce04524a880e997f2592e68 P 136c8b33bab647ab9bc8dce56ec46d12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63e3996d9f3d4b549e96bbb2d8a24f16" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 40773 } } peers { permanent_uuid: "b79952721ba14aa0bf5211cc3ce70548" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 33777 } } peers { permanent_uuid: "136c8b33bab647ab9bc8dce56ec46d12" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36177 } }
I20260926 06:16:41.251705 14007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee70ca81ce04524a880e997f2592e68" candidate_uuid: "136c8b33bab647ab9bc8dce56ec46d12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63e3996d9f3d4b549e96bbb2d8a24f16"
I20260926 06:16:41.251771 14007 raft_consensus.cc:3037] T dee70ca81ce04524a880e997f2592e68 P 63e3996d9f3d4b549e96bbb2d8a24f16 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:16:41.252573 14007 raft_consensus.cc:2454] T dee70ca81ce04524a880e997f2592e68 P 63e3996d9f3d4b549e96bbb2d8a24f16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 136c8b33bab647ab9bc8dce56ec46d12 in term 1.
I20260926 06:16:41.252745 14081 leader_election.cc:302] T dee70ca81ce04524a880e997f2592e68 P 136c8b33bab647ab9bc8dce56ec46d12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 136c8b33bab647ab9bc8dce56ec46d12, 63e3996d9f3d4b549e96bbb2d8a24f16; no voters:
I20260926 06:16:41.252925 14209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:16:41.245520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:44057 (local address 127.2.110.193:33777)
0926 06:16:41.245567 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 06:16:41.245569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:16:41.249025 (+ 3456us) server_negotiation.cc:407] Connection header received
0926 06:16:41.249055 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:16:41.249056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:16:41.249066 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:16:41.249111 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:16:41.249876 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:16:41.250582 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:16:41.251878 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:16:41.252460 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:16:41.252544 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:16:41.252554 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 06:16:41.252813 (+ 259us) server_negotiation.cc:299] Negotiation successful
0926 06:16:41.252857 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 06:16:41.252981 14194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:16:41.244803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.193:33777 (local address 127.2.110.195:44057)
0926 06:16:41.248918 (+ 4115us) negotiation.cc:107] Waiting for socket to connect
0926 06:16:41.248929 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:16:41.248969 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:16:41.249749 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:16:41.249750 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:16:41.249752 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:16:41.249788 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:16:41.249789 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:16:41.250729 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:16:41.250730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:16:41.251733 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:16:41.251736 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:16:41.252585 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:16:41.252586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:16:41.252642 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:16:41.252646 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:16:41.252693 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 06:16:41.252729 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4103,"thread_start_us":116,"threads_started":1}
I20260926 06:16:41.253295 14218 leader_election.cc:288] T dee70ca81ce04524a880e997f2592e68 P 136c8b33bab647ab9bc8dce56ec46d12 [CANDIDATE]: Term 1 election: Requested vote from peers 63e3996d9f3d4b549e96bbb2d8a24f16 (127.2.110.194:40773), b79952721ba14aa0bf5211cc3ce70548 (127.2.110.193:33777)
I20260926 06:16:41.253262 13869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee70ca81ce04524a880e997f2592e68" candidate_uuid: "136c8b33bab647ab9bc8dce56ec46d12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b79952721ba14aa0bf5211cc3ce70548" is_pre_election: true
I20260926 06:16:41.253369 14218 raft_consensus.cc:2781] T dee70ca81ce04524a880e997f2592e68 P 136c8b33bab647ab9bc8dce56ec46d12 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:16:41.253408 14218 raft_consensus.cc:686] T dee70ca81ce04524a880e997f2592e68 P 136c8b33bab647ab9bc8dce56ec46d12 [term 1 LEADER]: Becoming Leader. State: Replica: 136c8b33bab647ab9bc8dce56ec46d12, State: Running, Role: LEADER
I20260926 06:16:41.253454 13869 raft_consensus.cc:2454] T dee70ca81ce04524a880e997f2592e68 P b79952721ba14aa0bf5211cc3ce70548 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 136c8b33bab647ab9bc8dce56ec46d12 in term 0.
I20260926 06:16:41.253496 14218 consensus_queue.cc:237] T dee70ca81ce04524a880e997f2592e68 P 136c8b33bab647ab9bc8dce56ec46d12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63e3996d9f3d4b549e96bbb2d8a24f16" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 40773 } } peers { permanent_uuid: "b79952721ba14aa0bf5211cc3ce70548" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 33777 } } peers { permanent_uuid: "136c8b33bab647ab9bc8dce56ec46d12" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36177 } }
I20260926 06:16:41.253605 13870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee70ca81ce04524a880e997f2592e68" candidate_uuid: "136c8b33bab647ab9bc8dce56ec46d12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b79952721ba14aa0bf5211cc3ce70548"
I20260926 06:16:41.253679 13870 raft_consensus.cc:3037] T dee70ca81ce04524a880e997f2592e68 P b79952721ba14aa0bf5211cc3ce70548 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:16:41.254398 13707 catalog_manager.cc:5495] T dee70ca81ce04524a880e997f2592e68 P 136c8b33bab647ab9bc8dce56ec46d12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 136c8b33bab647ab9bc8dce56ec46d12 (127.2.110.195). New cstate: current_term: 1 leader_uuid: "136c8b33bab647ab9bc8dce56ec46d12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63e3996d9f3d4b549e96bbb2d8a24f16" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 40773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b79952721ba14aa0bf5211cc3ce70548" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 33777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "136c8b33bab647ab9bc8dce56ec46d12" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36177 } health_report { overall_health: HEALTHY } } }
I20260926 06:16:41.254606 13870 raft_consensus.cc:2454] T dee70ca81ce04524a880e997f2592e68 P b79952721ba14aa0bf5211cc3ce70548 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 136c8b33bab647ab9bc8dce56ec46d12 in term 1.
Sep 26 06:16:41 dist-test-slave-hndd krb5kdc[13232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403400, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.254@KRBTEST.COM for HTTP/127.2.110.212@KRBTEST.COM
I20260926 06:16:41.271759 14207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:16:41.268396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39072 (local address 127.2.110.195:36177)
0926 06:16:41.268438 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 06:16:41.268440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:16:41.268447 (+ 7us) server_negotiation.cc:407] Connection header received
0926 06:16:41.268484 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:16:41.268484 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:16:41.268517 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:16:41.268584 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:16:41.268802 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:16:41.269526 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:16:41.270592 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:16:41.270897 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:16:41.270963 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:16:41.271172 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:16:41.271305 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:16:41.271385 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 06:16:41.271633 (+ 248us) server_negotiation.cc:299] Negotiation successful
0926 06:16:41.271657 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 06:16:41.282599 13870 raft_consensus.cc:1261] T dee70ca81ce04524a880e997f2592e68 P b79952721ba14aa0bf5211cc3ce70548 [term 1 FOLLOWER]: Refusing update from remote peer 136c8b33bab647ab9bc8dce56ec46d12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:16:41.283120 14220 consensus_queue.cc:1035] T dee70ca81ce04524a880e997f2592e68 P 136c8b33bab647ab9bc8dce56ec46d12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b79952721ba14aa0bf5211cc3ce70548" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 33777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:16:41.285686 14007 raft_consensus.cc:1261] T dee70ca81ce04524a880e997f2592e68 P 63e3996d9f3d4b549e96bbb2d8a24f16 [term 1 FOLLOWER]: Refusing update from remote peer 136c8b33bab647ab9bc8dce56ec46d12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:16:41.286118 14220 consensus_queue.cc:1035] T dee70ca81ce04524a880e997f2592e68 P 136c8b33bab647ab9bc8dce56ec46d12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63e3996d9f3d4b549e96bbb2d8a24f16" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 40773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:16:41.288879 14224 mvcc.cc:205] Tried to move back new op lower bound from 7333492331650301952 to 7333492331535028224. Current Snapshot: MvccSnapshot[applied={T|T < 7333492331650301952}]
I20260926 06:16:41.289247 14223 mvcc.cc:205] Tried to move back new op lower bound from 7333492331650301952 to 7333492331535028224. Current Snapshot: MvccSnapshot[applied={T|T < 7333492331650301952}]
I20260926 06:16:41.291893 13707 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:16:41.291999 13707 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:16:41.292896 13707 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aec92f6a38b646a3843c07c440196b9a: Sending DeleteTablet for 3 replicas of tablet dee70ca81ce04524a880e997f2592e68
I20260926 06:16:41.293246 14125 tablet_service.cc:1514] Processing DeleteTablet for tablet dee70ca81ce04524a880e997f2592e68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:16:41 UTC) from {username='kudu', principal='kudu/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:39066
I20260926 06:16:41.293671 13842 tablet_service.cc:1514] Processing DeleteTablet for tablet dee70ca81ce04524a880e997f2592e68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:16:41 UTC) from {username='kudu', principal='kudu/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:54868
I20260926 06:16:41.294083 13992 tablet_service.cc:1514] Processing DeleteTablet for tablet dee70ca81ce04524a880e997f2592e68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:16:41 UTC) from {username='kudu', principal='kudu/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:55202
I20260926 06:16:41.294727 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 13789
I20260926 06:16:41.296939 14231 tablet_replica.cc:330] T dee70ca81ce04524a880e997f2592e68 P 136c8b33bab647ab9bc8dce56ec46d12: stopping tablet replica
I20260926 06:16:41.296957 14226 mvcc.cc:205] Tried to move back new op lower bound from 7333492331650301952 to 7333492331535028224. Current Snapshot: MvccSnapshot[applied={T|T < 7333492331650301952}]
I20260926 06:16:41.297046 14231 raft_consensus.cc:2229] T dee70ca81ce04524a880e997f2592e68 P 136c8b33bab647ab9bc8dce56ec46d12 [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:16:41.297153 14231 raft_consensus.cc:2258] T dee70ca81ce04524a880e997f2592e68 P 136c8b33bab647ab9bc8dce56ec46d12 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 06:16:41.304258 13693 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.110.193:33777 (error 108)
I20260926 06:16:41.304980 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 13926
I20260926 06:16:41.306517 14204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:16:41.306369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.193:33777 (local address 127.0.0.1:54880)
0926 06:16:41.306423 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 06:16:41.306460 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.193:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 06:16:41.306774 13693 catalog_manager.cc:4584] TS b79952721ba14aa0bf5211cc3ce70548 (127.2.110.193:33777): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dee70ca81ce04524a880e997f2592e68: Network error: Client connection negotiation failed: client connection to 127.2.110.193:33777: connect: Connection refused (error 111)
I20260926 06:16:41.311625 14231 ts_tablet_manager.cc:1905] T dee70ca81ce04524a880e997f2592e68 P 136c8b33bab647ab9bc8dce56ec46d12: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 06:16:41.311884 14204 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:16:41.311629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.194:40773 (local address 127.0.0.1:55218)
0926 06:16:41.311677 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 06:16:41.311688 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:16:41.311726 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:16:41.311831 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.194:40773: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 06:16:41.312316 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 14063
W20260926 06:16:41.321979 14204 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:16:41.321651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.195:36177 (local address 127.0.0.1:39084)
0926 06:16:41.321707 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 06:16:41.321718 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:16:41.321758 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:16:41.321927 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.195:36177: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 06:16:41.322535 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 13676
I20260926 06:16:41.328174 2491 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 06:16:41.663424 2491 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 06:16:41.663525 2491 mini_ranger.cc:66] Stopping Ranger...
I20260926 06:16:42.009045 2491 mini_ranger.cc:68] Stopped Ranger
2026-09-26 06:16:42.009 UTC [13262] LOG: received smart shutdown request
2026-09-26 06:16:42.013 UTC [13262] LOG: background worker "logical replication launcher" (PID 13271) exited with exit code 1
2026-09-26 06:16:42.013 UTC [13266] LOG: shutting down
2026-09-26 06:16:42.208 UTC [13262] LOG: database system is shut down
2026-09-26T06:16:42Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (50281 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:16:42 dist-test-slave-hndd krb5kdc[14244](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:16:42 dist-test-slave-hndd krb5kdc[14244](info): set up 2 sockets
Sep 26 06:16:42 dist-test-slave-hndd krb5kdc[14244](info): commencing operation
krb5kdc: starting...
W20260926 06:16:43.397639 2491 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Sep 26 06:16:43 dist-test-slave-hndd krb5kdc[14244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:16:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:16:43Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.110.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.110.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.110.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/rangeradmin_127.2.110.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.110.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/rangeradmin_127.2.110.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.110.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.110.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.110.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/rangerlookup_127.2.110.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.110.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/rangerlookup_127.2.110.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.110.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.110.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.110.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/HTTP_127.2.110.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.110.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/HTTP_127.2.110.212@KRBTEST.COM.keytab.
I20260926 06:16:43.576350 2491 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskRgcQD_/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/postgres -l logfile start
2026-09-26 06:16:46.212 UTC [14274] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 06:16:46.213 UTC [14274] LOG: listening on IPv4 address "127.2.110.212", port 49631
2026-09-26 06:16:46.217 UTC [14274] LOG: listening on Unix socket "/tmp/.s.PGSQL.49631"
2026-09-26 06:16:46.240 UTC [14277] LOG: database system was shut down at 2026-09-26 06:16:44 UTC
2026-09-26 06:16:46.244 UTC [14274] LOG: database system is ready to accept connections
I20260926 06:16:47.209260 2491 mini_postgres.cc:96] Postgres bound to 49631
2026-09-26 06:16:47.215 UTC [14285] FATAL: database "slave" does not exist
127.2.110.212:49631 - accepting connections
I20260926 06:16:47.216238 2491 mini_ranger.cc:161] Starting Ranger...
I20260926 06:16:47.231884 2491 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 06:16:47.756035 2491 mini_ranger.cc:87] Created ranger Postgres database
I20260926 06:16:47.756129 2491 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ranger-admin
2026-09-26 06:16:48,046 [I] DB FLAVOR :POSTGRES
2026-09-26 06:16:48,046 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 06:16:48,046 [I] Checking connection..
2026-09-26 06:16:48,047 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgcQD_/build/release/bin/postgresql.jar:/tmp/dist-test-taskRgcQD_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.110.212:49631/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 06:16:48,341 [I] Checking connection passed.
2026-09-26 06:16:48,342 [I] --------- Verifying version history table ---------
2026-09-26 06:16:48,342 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgcQD_/build/release/bin/postgresql.jar:/tmp/dist-test-taskRgcQD_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.110.212:49631/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 06:16:48,596 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 06:16:48,596 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgcQD_/build/release/bin/postgresql.jar:/tmp/dist-test-taskRgcQD_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.110.212:49631/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 06:16:48,853 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 06:16:48,854 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 06:16:48,854 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgcQD_/build/release/bin/postgresql.jar:/tmp/dist-test-taskRgcQD_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.110.212:49631/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRgcQD_/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 06:16:49.112 UTC [14349] WARNING: there is no transaction in progress
2026-09-26 06:16:49,121 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 06:16:49,121 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgcQD_/build/release/bin/postgresql.jar:/tmp/dist-test-taskRgcQD_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.110.212:49631/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 06:16:49,410 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 06:16:49,410 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 06:16:49,410 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgcQD_/build/release/bin/postgresql.jar:/tmp/dist-test-taskRgcQD_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.110.212:49631/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 06:16:49,679 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgcQD_/build/release/bin/postgresql.jar:/tmp/dist-test-taskRgcQD_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.110.212:49631/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 06:16:49,939 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgcQD_/build/release/bin/postgresql.jar:/tmp/dist-test-taskRgcQD_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.110.212:49631/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-hndd.c.gcp-upstream.internal','N') ;"
2026-09-26 06:16:50,201 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgcQD_/build/release/bin/postgresql.jar:/tmp/dist-test-taskRgcQD_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.110.212:49631/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 06:16:50,485 [I] Table x_portal_user does not exist in database ranger
2026-09-26 06:16:50,486 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgcQD_/build/release/bin/postgresql.jar:/tmp/dist-test-taskRgcQD_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.110.212:49631/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 06:16:50,757 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 06:16:50,758 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgcQD_/build/release/bin/postgresql.jar:/tmp/dist-test-taskRgcQD_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.110.212:49631/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 06:16:51,012 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgcQD_/build/release/bin/postgresql.jar:/tmp/dist-test-taskRgcQD_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.110.212:49631/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 06:16:51,286 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 06:16:51,286 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgcQD_/build/release/bin/postgresql.jar:/tmp/dist-test-taskRgcQD_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.110.212:49631/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRgcQD_/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 06:16:52.358 UTC [14457] WARNING: there is no transaction in progress
2026-09-26 06:16:52.376 UTC [14457] WARNING: there is no transaction in progress
2026-09-26 06:16:52.395 UTC [14457] WARNING: there is no transaction in progress
2026-09-26 06:16:52.412 UTC [14457] WARNING: there is no transaction in progress
2026-09-26 06:16:52.429 UTC [14457] WARNING: there is no transaction in progress
2026-09-26 06:16:52.448 UTC [14457] WARNING: there is no transaction in progress
2026-09-26 06:16:52.508 UTC [14457] WARNING: there is no transaction in progress
2026-09-26 06:16:52.518 UTC [14457] WARNING: there is no transaction in progress
2026-09-26 06:16:52.534 UTC [14457] WARNING: there is no transaction in progress
2026-09-26 06:16:52.545 UTC [14457] WARNING: there is no transaction in progress
2026-09-26 06:16:52.556 UTC [14457] WARNING: there is no transaction in progress
2026-09-26 06:16:52.570 UTC [14457] WARNING: there is no transaction in progress
2026-09-26 06:16:52.581 UTC [14457] WARNING: there is no transaction in progress
2026-09-26 06:16:52.589 UTC [14457] WARNING: there is no transaction in progress
2026-09-26 06:16:52.596 UTC [14457] WARNING: there is no transaction in progress
2026-09-26 06:16:52.603 UTC [14457] WARNING: there is no transaction in progress
2026-09-26 06:16:53,361 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 06:16:53,362 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgcQD_/build/release/bin/postgresql.jar:/tmp/dist-test-taskRgcQD_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.110.212:49631/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 06:16:53,601 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 06:16:53,601 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgcQD_/build/release/bin/postgresql.jar:/tmp/dist-test-taskRgcQD_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.110.212:49631/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 06:16:53,869 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 06:16:53,870 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgcQD_/build/release/bin/postgresql.jar:/tmp/dist-test-taskRgcQD_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.110.212:49631/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 06:16:54,127 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 06:16:54,128 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgcQD_/build/release/bin/postgresql.jar:/tmp/dist-test-taskRgcQD_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.110.212:49631/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 06:16:54,367 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgcQD_/build/release/bin/postgresql.jar:/tmp/dist-test-taskRgcQD_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.110.212:49631/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 06:16:54,651 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgcQD_/build/release/bin/postgresql.jar:/tmp/dist-test-taskRgcQD_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.110.212:49631/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-hndd.c.gcp-upstream.internal';"
2026-09-26 06:16:54,910 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 06:16:54,911 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgcQD_/build/release/bin/postgresql.jar:/tmp/dist-test-taskRgcQD_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.110.212:49631/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 06:16:55,184 [I] DB_PATCHES have already been applied
I20260926 06:16:55.191391 2491 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ranger-admin:/tmp/dist-test-taskRgcQD_/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskRgcQD_/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskRgcQD_/build/release/bin/postgresql.jar:/tmp/dist-test-taskRgcQD_/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 06:16:55.191468 2491 mini_ranger.cc:193] Using host: 127.2.110.212
I20260926 06:16:55.194768 2491 mini_ranger.cc:227] Ranger admin URL: http://127.2.110.212:42235
Sep 26, 2026 6:16:55 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 6:16:55 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 6:16:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 6:16:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 6:16:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 6:16:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 6:16:56 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 06:16:56.378662 2491 mini_ranger.cc:160] Time spent starting Ranger: real 9.162s user 0.000s sys 0.011s
Sep 26, 2026 6:17:19 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 6:17:19 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 6:17:19 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 6:17:19 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 6:17:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 6:17:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 6:17:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 6:17:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 6:17:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 6:17:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 6:17:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 6:17:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 6:17:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 6:17:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 6:17:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 6:17:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 6:17:19 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 6:17:20 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 06:17:21.586561 2491 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.110.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.110.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.110.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/rangerkms_127.2.110.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.110.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/rangerkms_127.2.110.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.110.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.110.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.110.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/HTTP_127.2.110.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.110.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/HTTP_127.2.110.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 06:17:21 dist-test-slave-hndd krb5kdc[14244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403441, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 06:17:21.679524 2491 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 06:17:21.689579 2491 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 06:17:22.710937 2491 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 06:17:22.711050 2491 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ranger-kms
I20260926 06:17:22.711076 2491 mini_ranger_kms.cc:227] Using postgres at 127.2.110.212:49631
2026-09-26 06:17:22,895 [I] DB FLAVOR :POSTGRES
2026-09-26 06:17:22,895 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 06:17:22,895 [I] Checking connection
2026-09-26 06:17:22,895 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgcQD_/build/release/bin/postgresql.jar:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.110.212:49631/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 06:17:23,163 [I] connection success
2026-09-26 06:17:23,164 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 06:17:23,164 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 06:17:23,164 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgcQD_/build/release/bin/postgresql.jar:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.110.212:49631/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 06:17:23,439 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 06:17:23,439 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 06:17:23,439 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 06:17:23,439 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgcQD_/build/release/bin/postgresql.jar:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.110.212:49631/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 06:17:23,717 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 06:17:24.222751 2491 mini_ranger_kms.cc:310] Created kms service
I20260926 06:17:24.436993 2491 mini_ranger_kms.cc:326] Created kudu user
I20260926 06:17:24.487408 2491 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 06:17:24 dist-test-slave-hndd krb5kdc[14244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403441, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.110.212@KRBTEST.COM
I20260926 06:17:24.960510 2491 mini_ranger_kms.cc:384] Added ranger policy
I20260926 06:17:24.960747 2491 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ranger-kms:/tmp/dist-test-taskRgcQD_/build/release/bin/postgresql.jar:/tmp/dist-test-taskRgcQD_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskRgcQD_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 06:17:24.960770 2491 mini_ranger_kms.cc:240] Using host: 127.2.110.212
I20260926 06:17:24.964342 2491 mini_ranger_kms.cc:276] Ranger KMS PID: 14638
I20260926 06:17:24.964450 2491 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.110.212:41919
Sep 26, 2026 6:17:25 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 6:17:25 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 6:17:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 6:17:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 6:17:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 6:17:26 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 06:17:27.218014 2491 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.538s user 0.000s sys 0.166s
I20260926 06:17:27.218111 2491 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 06:17:27.218158 2491 mini_ranger_kms.cc:401] 127.2.110.212:41919/kms/v1/keys
Sep 26 06:17:27 dist-test-slave-hndd krb5kdc[14244](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790403447, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.110.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 06:17:27 dist-test-slave-hndd krb5kdc[14244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403447, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.110.212@KRBTEST.COM for HTTP/127.2.110.212@KRBTEST.COM
Sep 26 06:17:28 dist-test-slave-hndd krb5kdc[14244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403447, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.110.212@KRBTEST.COM for HTTP/127.2.110.212@KRBTEST.COM
Sep 26 06:17:30 dist-test-slave-hndd krb5kdc[14244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
I20260926 06:17:30.598464 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:35959
--webserver_interface=127.2.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:41123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.110.254:35959
--ranger_config_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.110.212:41919/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:17:30.690224 14688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:17:30.690497 14688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:17:30.690541 14688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:17:30.691826 14688 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:17:30.691903 14688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:17:30.691931 14688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:17:30.691951 14688 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:17:30.691982 14688 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:17:30.693672 14688 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:41123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.110.212:41919/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.110.254:35959
--ranger_config_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:35959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:17:30.693959 14688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:17:30.694200 14688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:17:30.699832 14694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:17:30.699958 14693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:17:30.699959 14688 server_base.cc:1034] running on GCE node
W20260926 06:17:30.699908 14696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:17:30.700488 14688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:17:30.701025 14688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:17:30.702237 14688 hybrid_clock.cc:648] HybridClock initialized: now 1790403450702209 us; error 78 us; skew 500 ppm
Sep 26 06:17:30 dist-test-slave-hndd krb5kdc[14244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403450, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:17:30.705181 14688 init.cc:377] Logged in from keytab as kudu/127.2.110.254@KRBTEST.COM (short username kudu)
I20260926 06:17:30.705734 14688 webserver.cc:460] Webserver started at http://127.2.110.254:45077/ using document root <none> and password file <none>
I20260926 06:17:30.705976 14688 fs_manager.cc:359] Metadata directory not provided
I20260926 06:17:30.706038 14688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:17:30.706212 14688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 06:17:30 dist-test-slave-hndd krb5kdc[14244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403450, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.254@KRBTEST.COM for HTTP/127.2.110.212@KRBTEST.COM
I20260926 06:17:30.762637 14688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/master-0/data/instance:
uuid: "eb2e6df8b9ce4393bbb8e7f8dd777b37"
format_stamp: "Formatted at 2026-09-26 06:17:30 on dist-test-slave-hndd"
server_key: "ec7518fb7ddb0eada7dd0c566c196cb3"
server_key_iv: "b053c1fa670fb2c15e914871140a90a5"
server_key_version: "kuduclusterkey@0"
I20260926 06:17:30.763119 14688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/master-0/wal/instance:
uuid: "eb2e6df8b9ce4393bbb8e7f8dd777b37"
format_stamp: "Formatted at 2026-09-26 06:17:30 on dist-test-slave-hndd"
server_key: "ec7518fb7ddb0eada7dd0c566c196cb3"
server_key_iv: "b053c1fa670fb2c15e914871140a90a5"
server_key_version: "kuduclusterkey@0"
I20260926 06:17:30.764621 14688 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:17:30.777096 14703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:17:30.777324 14688 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:17:30.777546 14688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "eb2e6df8b9ce4393bbb8e7f8dd777b37"
format_stamp: "Formatted at 2026-09-26 06:17:30 on dist-test-slave-hndd"
server_key: "ec7518fb7ddb0eada7dd0c566c196cb3"
server_key_iv: "b053c1fa670fb2c15e914871140a90a5"
server_key_version: "kuduclusterkey@0"
I20260926 06:17:30.777696 14688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:17:30.787495 14688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:17:30.789599 14688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:17:30.790014 14688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:17:30.794865 14688 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.254:35959
I20260926 06:17:30.794950 14755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.254:35959 every 8 connection(s)
I20260926 06:17:30.795442 14688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:17:30.796473 14756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:17:30.799624 14756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb2e6df8b9ce4393bbb8e7f8dd777b37: Bootstrap starting.
I20260926 06:17:30.800633 14756 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb2e6df8b9ce4393bbb8e7f8dd777b37: Neither blocks nor log segments found. Creating new log.
I20260926 06:17:30.801208 14756 log.cc:824] T 00000000000000000000000000000000 P eb2e6df8b9ce4393bbb8e7f8dd777b37: Log is configured to *not* fsync() on all Append() calls
I20260926 06:17:30.802323 14756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb2e6df8b9ce4393bbb8e7f8dd777b37: No bootstrap required, opened a new log
I20260926 06:17:30.805320 14756 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb2e6df8b9ce4393bbb8e7f8dd777b37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb2e6df8b9ce4393bbb8e7f8dd777b37" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 35959 } }
I20260926 06:17:30.805526 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 14688
I20260926 06:17:30.805572 14756 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb2e6df8b9ce4393bbb8e7f8dd777b37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:17:30.805608 14756 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb2e6df8b9ce4393bbb8e7f8dd777b37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2e6df8b9ce4393bbb8e7f8dd777b37, State: Initialized, Role: FOLLOWER
I20260926 06:17:30.805631 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/master-0/wal/instance
I20260926 06:17:30.805830 2491 external_mini_cluster.cc:1428] Setting key c65f32d157f124878df7267c46334699
I20260926 06:17:30.805806 14756 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb2e6df8b9ce4393bbb8e7f8dd777b37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb2e6df8b9ce4393bbb8e7f8dd777b37" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 35959 } }
I20260926 06:17:30.805881 14756 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb2e6df8b9ce4393bbb8e7f8dd777b37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:17:30.805944 14756 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb2e6df8b9ce4393bbb8e7f8dd777b37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:17:30.806005 14756 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb2e6df8b9ce4393bbb8e7f8dd777b37 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:17:30.806835 14756 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb2e6df8b9ce4393bbb8e7f8dd777b37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb2e6df8b9ce4393bbb8e7f8dd777b37" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 35959 } }
I20260926 06:17:30.807062 14756 leader_election.cc:302] T 00000000000000000000000000000000 P eb2e6df8b9ce4393bbb8e7f8dd777b37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb2e6df8b9ce4393bbb8e7f8dd777b37; no voters:
I20260926 06:17:30.807257 14756 leader_election.cc:288] T 00000000000000000000000000000000 P eb2e6df8b9ce4393bbb8e7f8dd777b37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:17:30.807299 14761 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb2e6df8b9ce4393bbb8e7f8dd777b37 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:17:30.807538 14761 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb2e6df8b9ce4393bbb8e7f8dd777b37 [term 1 LEADER]: Becoming Leader. State: Replica: eb2e6df8b9ce4393bbb8e7f8dd777b37, State: Running, Role: LEADER
I20260926 06:17:30.807627 14756 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb2e6df8b9ce4393bbb8e7f8dd777b37 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:17:30.807699 14761 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb2e6df8b9ce4393bbb8e7f8dd777b37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb2e6df8b9ce4393bbb8e7f8dd777b37" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 35959 } }
I20260926 06:17:30.808779 14763 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb2e6df8b9ce4393bbb8e7f8dd777b37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb2e6df8b9ce4393bbb8e7f8dd777b37. Latest consensus state: current_term: 1 leader_uuid: "eb2e6df8b9ce4393bbb8e7f8dd777b37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb2e6df8b9ce4393bbb8e7f8dd777b37" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 35959 } } }
I20260926 06:17:30.808856 14763 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb2e6df8b9ce4393bbb8e7f8dd777b37 [sys.catalog]: This master's current role is: LEADER
I20260926 06:17:30.809144 14756 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 06:17:30.809239 14762 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb2e6df8b9ce4393bbb8e7f8dd777b37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb2e6df8b9ce4393bbb8e7f8dd777b37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb2e6df8b9ce4393bbb8e7f8dd777b37" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 35959 } } }
I20260926 06:17:30.809305 14762 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb2e6df8b9ce4393bbb8e7f8dd777b37 [sys.catalog]: This master's current role is: LEADER
I20260926 06:17:30.809437 14775 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:17:30.813163 14775 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:17:30.816749 14775 catalog_manager.cc:1347] Generated new cluster ID: 38198e1edd474deb8321df7dd1f2ab09
I20260926 06:17:30.816817 14775 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 06:17:30 dist-test-slave-hndd krb5kdc[14244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:17:30.823771 14760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:30.806352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42976 (local address 127.2.110.254:35959)
0926 06:17:30.806561 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 06:17:30.806572 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 06:17:30.806641 (+ 69us) server_negotiation.cc:407] Connection header received
0926 06:17:30.806754 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:17:30.806760 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:17:30.806808 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:17:30.806957 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:17:30.807795 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:30.809241 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:30.811320 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:30.813573 (+ 2253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:30.813756 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:30.817850 (+ 4094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:17:30.817858 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:17:30.817859 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:17:30.817878 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:17:30.821394 (+ 3516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:30.821983 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:30.821988 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:30.821988 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:30.822123 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:30.822327 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:30.822328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:30.822328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:30.822670 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:17:30.822807 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 06:17:30.823282 (+ 475us) server_negotiation.cc:299] Negotiation successful
0926 06:17:30.823451 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":82,"threads_started":1}
I20260926 06:17:30.825013 14775 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:17:30.825831 14775 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:17:30.836478 14775 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb2e6df8b9ce4393bbb8e7f8dd777b37: Generated new TSK 0
I20260926 06:17:30.836758 14775 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab.
Entry for principal kudu/127.2.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab.
I20260926 06:17:31.597287 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.110.212:41919/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.193:0
--local_ip_for_outbound_sockets=127.2.110.193
--webserver_interface=127.2.110.193
--webserver_port=0
--tserver_master_addrs=127.2.110.254:35959
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab
--principal=kudu/127.2.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:41123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:17:31.722333 14802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:17:31.722656 14802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:17:31.722694 14802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:17:31.724576 14802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:17:31.724668 14802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:17:31.724714 14802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.193
I20260926 06:17:31.727120 14802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:41123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.110.212:41919/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab
--principal=kudu/127.2.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.110.193
--webserver_port=0
--tserver_master_addrs=127.2.110.254:35959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.193
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:17:31.727420 14802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:17:31.727748 14802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 06:17:31.734503 14802 server_base.cc:1034] running on GCE node
W20260926 06:17:31.734400 14808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:17:31.734338 14807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:17:31.734745 14810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:17:31.735154 14802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:17:31.735436 14802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:17:31.736634 14802 hybrid_clock.cc:648] HybridClock initialized: now 1790403451736614 us; error 34 us; skew 500 ppm
Sep 26 06:17:31 dist-test-slave-hndd krb5kdc[14244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403451, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:17:31.740557 14802 init.cc:377] Logged in from keytab as kudu/127.2.110.193@KRBTEST.COM (short username kudu)
I20260926 06:17:31.741114 14802 webserver.cc:460] Webserver started at http://127.2.110.193:41039/ using document root <none> and password file <none>
I20260926 06:17:31.741477 14802 fs_manager.cc:359] Metadata directory not provided
I20260926 06:17:31.741572 14802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:17:31.741750 14802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 06:17:31 dist-test-slave-hndd krb5kdc[14244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403451, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.193@KRBTEST.COM for HTTP/127.2.110.212@KRBTEST.COM
I20260926 06:17:31.758217 14802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-0/data/instance:
uuid: "61a97084ca5945f889a2aa4893685ed5"
format_stamp: "Formatted at 2026-09-26 06:17:31 on dist-test-slave-hndd"
server_key: "c832b9aa050945aaee53f182168acae5"
server_key_iv: "57aa8078a6ff3974262f308ff3957ad1"
server_key_version: "kuduclusterkey@0"
I20260926 06:17:31.758817 14802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-0/wal/instance:
uuid: "61a97084ca5945f889a2aa4893685ed5"
format_stamp: "Formatted at 2026-09-26 06:17:31 on dist-test-slave-hndd"
server_key: "c832b9aa050945aaee53f182168acae5"
server_key_iv: "57aa8078a6ff3974262f308ff3957ad1"
server_key_version: "kuduclusterkey@0"
I20260926 06:17:31.763509 14802 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.000s
I20260926 06:17:31.776508 14817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:17:31.776769 14802 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:17:31.776892 14802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-0/wal
uuid: "61a97084ca5945f889a2aa4893685ed5"
format_stamp: "Formatted at 2026-09-26 06:17:31 on dist-test-slave-hndd"
server_key: "c832b9aa050945aaee53f182168acae5"
server_key_iv: "57aa8078a6ff3974262f308ff3957ad1"
server_key_version: "kuduclusterkey@0"
I20260926 06:17:31.777006 14802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:17:31.796255 14802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:17:31.798753 14802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:17:31.798902 14802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:17:31.799188 14802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:17:31.799587 14802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:17:31.799625 14802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:17:31.799647 14802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:17:31.799691 14802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:17:31.808161 14802 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.193:40231
I20260926 06:17:31.808462 14932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.193:40231 every 8 connection(s)
I20260926 06:17:31.808959 14802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
Sep 26 06:17:31 dist-test-slave-hndd krb5kdc[14244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403451, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.193@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:17:31.818135 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 14802
I20260926 06:17:31.818261 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-0/wal/instance
I20260926 06:17:31.818761 2491 external_mini_cluster.cc:1428] Setting key e21893802f236f80c479dba83ca0e0cf
I20260926 06:17:31.823755 14935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:31.809845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35959 (local address 127.2.110.193:49893)
0926 06:17:31.810240 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:31.810291 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 06:17:31.810945 (+ 654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:17:31.811354 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:17:31.811358 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:17:31.811614 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:17:31.811906 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:31.811912 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:31.812792 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:31.812793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:31.814292 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:31.814295 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:31.814753 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:31.814754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:31.814895 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:31.815543 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:17:31.815554 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:17:31.817133 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:17:31.819879 (+ 2746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:17:31.819892 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:17:31.819895 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:17:31.820551 (+ 656us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:17:31.820956 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:17:31.820958 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:17:31.820958 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:17:31.821141 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:17:31.821550 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:17:31.821559 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:17:31.822684 (+ 1125us) client_negotiation.cc:769] Sending connection context
0926 06:17:31.823140 (+ 456us) client_negotiation.cc:240] Negotiation successful
0926 06:17:31.823434 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":85,"threads_started":1}
I20260926 06:17:31.824550 14936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:31.810007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:49893 (local address 127.2.110.254:35959)
0926 06:17:31.810163 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 06:17:31.810166 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:17:31.810411 (+ 245us) server_negotiation.cc:407] Connection header received
0926 06:17:31.811135 (+ 724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:17:31.811136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:17:31.811152 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:17:31.811219 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:17:31.812040 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:31.812679 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:31.814408 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:31.814631 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:31.814725 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:31.817254 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:17:31.817256 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:17:31.817256 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:17:31.817265 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:17:31.819654 (+ 2389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:31.820715 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:31.820718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:31.820718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:31.820869 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:31.821214 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:31.821215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:31.821215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:31.821444 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:17:31.821516 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 06:17:31.824271 (+ 2755us) server_negotiation.cc:299] Negotiation successful
0926 06:17:31.824399 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":80,"threads_started":1}
I20260926 06:17:31.825413 14933 heartbeater.cc:347] Connected to a master server at 127.2.110.254:35959
I20260926 06:17:31.825520 14933 heartbeater.cc:464] Registering TS with master...
I20260926 06:17:31.825748 14933 heartbeater.cc:511] Master 127.2.110.254:35959 requested a full tablet report, sending...
I20260926 06:17:31.826399 14720 ts_manager.cc:194] Registered new tserver with Master: 61a97084ca5945f889a2aa4893685ed5 (127.2.110.193:40231)
I20260926 06:17:31.827059 14720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.193@KRBTEST.COM'} at 127.2.110.193:49893
WARNING: no policy specified for kudu/127.2.110.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab.
Entry for principal kudu/127.2.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab.
I20260926 06:17:31.852563 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.110.212:41919/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.194:0
--local_ip_for_outbound_sockets=127.2.110.194
--webserver_interface=127.2.110.194
--webserver_port=0
--tserver_master_addrs=127.2.110.254:35959
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab
--principal=kudu/127.2.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:41123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:17:31.961889 14939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:17:31.962136 14939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:17:31.962176 14939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:17:31.963383 14939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:17:31.963456 14939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:17:31.963505 14939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.194
I20260926 06:17:31.965114 14939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:41123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.110.212:41919/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab
--principal=kudu/127.2.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.110.194
--webserver_port=0
--tserver_master_addrs=127.2.110.254:35959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.194
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:17:31.965427 14939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:17:31.965703 14939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:17:31.970893 14947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:17:31.970861 14949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:17:31.971136 14946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:17:31.971971 14939 server_base.cc:1034] running on GCE node
I20260926 06:17:31.972190 14939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:17:31.972477 14939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:17:31.973686 14939 hybrid_clock.cc:648] HybridClock initialized: now 1790403451973650 us; error 36 us; skew 500 ppm
Sep 26 06:17:31 dist-test-slave-hndd krb5kdc[14244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403451, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:17:31.977895 14939 init.cc:377] Logged in from keytab as kudu/127.2.110.194@KRBTEST.COM (short username kudu)
I20260926 06:17:31.978482 14939 webserver.cc:460] Webserver started at http://127.2.110.194:37527/ using document root <none> and password file <none>
I20260926 06:17:31.978798 14939 fs_manager.cc:359] Metadata directory not provided
I20260926 06:17:31.978874 14939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:17:31.979023 14939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 06:17:31 dist-test-slave-hndd krb5kdc[14244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403451, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.194@KRBTEST.COM for HTTP/127.2.110.212@KRBTEST.COM
I20260926 06:17:31.993000 14939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-1/data/instance:
uuid: "5131ae7792e44f7f974f659ae02595da"
format_stamp: "Formatted at 2026-09-26 06:17:31 on dist-test-slave-hndd"
server_key: "36ec4ede4483576d6cb2423d7dceef88"
server_key_iv: "7a442223b181d1b346a965f3e2779909"
server_key_version: "kuduclusterkey@0"
I20260926 06:17:31.993527 14939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-1/wal/instance:
uuid: "5131ae7792e44f7f974f659ae02595da"
format_stamp: "Formatted at 2026-09-26 06:17:31 on dist-test-slave-hndd"
server_key: "36ec4ede4483576d6cb2423d7dceef88"
server_key_iv: "7a442223b181d1b346a965f3e2779909"
server_key_version: "kuduclusterkey@0"
I20260926 06:17:31.995532 14939 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260926 06:17:32.010375 14956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:17:32.010640 14939 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.000s
I20260926 06:17:32.010736 14939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-1/wal
uuid: "5131ae7792e44f7f974f659ae02595da"
format_stamp: "Formatted at 2026-09-26 06:17:31 on dist-test-slave-hndd"
server_key: "36ec4ede4483576d6cb2423d7dceef88"
server_key_iv: "7a442223b181d1b346a965f3e2779909"
server_key_version: "kuduclusterkey@0"
I20260926 06:17:32.010835 14939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 26 06:17:32 dist-test-slave-hndd krb5kdc[14244](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790403452, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:17:32.026659 14939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:17:32.029302 14939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:17:32.029493 14939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:17:32.029732 14939 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:17:32.030128 14939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:17:32.030167 14939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:17:32.030200 14939 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:17:32.030220 14939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:17:32.036827 14939 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.194:43281
I20260926 06:17:32.037382 14939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
I20260926 06:17:32.041236 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 14939
I20260926 06:17:32.041390 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-1/wal/instance
I20260926 06:17:32.041607 2491 external_mini_cluster.cc:1428] Setting key 1cc664f46ea97d474698681757e4c5a2
I20260926 06:17:32.049984 15069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.194:43281 every 8 connection(s)
WARNING: no policy specified for kudu/127.2.110.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.195@KRBTEST.COM" created.
Sep 26 06:17:32 dist-test-slave-hndd krb5kdc[14244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403451, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.194@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:17:32.080899 14936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:32.050572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:57731 (local address 127.2.110.254:35959)
0926 06:17:32.050658 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 06:17:32.050661 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:17:32.059089 (+ 8428us) server_negotiation.cc:407] Connection header received
0926 06:17:32.059995 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:17:32.059997 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:17:32.060016 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:17:32.060052 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:17:32.061051 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:32.061929 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:32.070569 (+ 8640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:32.071003 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:32.071061 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:32.076727 (+ 5666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:17:32.076729 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:17:32.076730 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:17:32.076740 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:17:32.078776 (+ 2036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:32.079403 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:32.079404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:32.079405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:32.079473 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:32.079806 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:32.079807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:32.079807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:32.079972 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:17:32.080102 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 06:17:32.080624 (+ 522us) server_negotiation.cc:299] Negotiation successful
0926 06:17:32.080741 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 06:17:32.081722 15073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:32.050221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35959 (local address 127.2.110.194:57731)
0926 06:17:32.058840 (+ 8619us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:32.058919 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0926 06:17:32.059827 (+ 908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:17:32.060276 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:17:32.060281 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:17:32.060594 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:17:32.060915 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:32.060921 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:32.062074 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:32.062075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:32.063470 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:32.063475 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:32.071333 (+ 7858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:32.071334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:32.073891 (+ 2557us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:32.074621 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:17:32.074633 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:17:32.076557 (+ 1924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:17:32.078907 (+ 2350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:17:32.078916 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:17:32.078919 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:17:32.079290 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:17:32.079558 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:17:32.079559 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:17:32.079559 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:17:32.079717 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:17:32.080148 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:17:32.080157 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:17:32.080433 (+ 276us) client_negotiation.cc:769] Sending connection context
0926 06:17:32.081026 (+ 593us) client_negotiation.cc:240] Negotiation successful
0926 06:17:32.081473 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8548,"thread_start_us":120,"threads_started":1}
I20260926 06:17:32.082310 15070 heartbeater.cc:347] Connected to a master server at 127.2.110.254:35959
I20260926 06:17:32.082422 15070 heartbeater.cc:464] Registering TS with master...
I20260926 06:17:32.082644 15070 heartbeater.cc:511] Master 127.2.110.254:35959 requested a full tablet report, sending...
I20260926 06:17:32.083007 14720 ts_manager.cc:194] Registered new tserver with Master: 5131ae7792e44f7f974f659ae02595da (127.2.110.194:43281)
I20260926 06:17:32.083745 14720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.194@KRBTEST.COM'} at 127.2.110.194:57731
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab.
Entry for principal kudu/127.2.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab.
I20260926 06:17:32.096911 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.110.212:41919/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.195:0
--local_ip_for_outbound_sockets=127.2.110.195
--webserver_interface=127.2.110.195
--webserver_port=0
--tserver_master_addrs=127.2.110.254:35959
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab
--principal=kudu/127.2.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:41123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:17:32.235360 15075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:17:32.235780 15075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:17:32.235875 15075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:17:32.237900 15075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:17:32.238035 15075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:17:32.238127 15075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.195
I20260926 06:17:32.241199 15075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:41123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.110.212:41919/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab
--principal=kudu/127.2.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.110.195
--webserver_port=0
--tserver_master_addrs=127.2.110.254:35959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.195
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:17:32.242116 15075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:17:32.242480 15075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:17:32.249194 15084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:17:32.249552 15082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:17:32.249788 15081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:17:32.252358 15075 server_base.cc:1034] running on GCE node
I20260926 06:17:32.252746 15075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:17:32.253118 15075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:17:32.260643 15075 hybrid_clock.cc:648] HybridClock initialized: now 1790403452260577 us; error 49 us; skew 500 ppm
Sep 26 06:17:32 dist-test-slave-hndd krb5kdc[14244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403452, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:17:32.265957 15075 init.cc:377] Logged in from keytab as kudu/127.2.110.195@KRBTEST.COM (short username kudu)
I20260926 06:17:32.266664 15075 webserver.cc:460] Webserver started at http://127.2.110.195:33259/ using document root <none> and password file <none>
I20260926 06:17:32.267009 15075 fs_manager.cc:359] Metadata directory not provided
I20260926 06:17:32.267158 15075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:17:32.267346 15075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 06:17:32 dist-test-slave-hndd krb5kdc[14244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403452, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.195@KRBTEST.COM for HTTP/127.2.110.212@KRBTEST.COM
I20260926 06:17:32.290586 15075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-2/data/instance:
uuid: "22a2a137f0f34c4590acc11d8139de50"
format_stamp: "Formatted at 2026-09-26 06:17:32 on dist-test-slave-hndd"
server_key: "530eaa174fd207176d40a60e8ea40d1e"
server_key_iv: "cb2a536319c8cfbc21670e4ec9c2558b"
server_key_version: "kuduclusterkey@0"
I20260926 06:17:32.291216 15075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-2/wal/instance:
uuid: "22a2a137f0f34c4590acc11d8139de50"
format_stamp: "Formatted at 2026-09-26 06:17:32 on dist-test-slave-hndd"
server_key: "530eaa174fd207176d40a60e8ea40d1e"
server_key_iv: "cb2a536319c8cfbc21670e4ec9c2558b"
server_key_version: "kuduclusterkey@0"
I20260926 06:17:32.294162 15075 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.001s
I20260926 06:17:32.315675 15091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:17:32.315953 15075 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 06:17:32.316048 15075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-2/wal
uuid: "22a2a137f0f34c4590acc11d8139de50"
format_stamp: "Formatted at 2026-09-26 06:17:32 on dist-test-slave-hndd"
server_key: "530eaa174fd207176d40a60e8ea40d1e"
server_key_iv: "cb2a536319c8cfbc21670e4ec9c2558b"
server_key_version: "kuduclusterkey@0"
I20260926 06:17:32.316145 15075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:17:32.334936 15075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:17:32.337484 15075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:17:32.337724 15075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:17:32.337994 15075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:17:32.338408 15075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:17:32.338500 15075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:17:32.338572 15075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:17:32.338629 15075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:17:32.345314 15075 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.195:42875
I20260926 06:17:32.346685 15075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
I20260926 06:17:32.347906 15204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.195:42875 every 8 connection(s)
I20260926 06:17:32.352939 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 15075
I20260926 06:17:32.353088 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-2/wal/instance
I20260926 06:17:32.353293 2491 external_mini_cluster.cc:1428] Setting key 7924803d65f82d3d476a8c24a48e2734
Sep 26 06:17:32 dist-test-slave-hndd krb5kdc[14244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403452, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.195@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:17:32.358291 14936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:32.348669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:43533 (local address 127.2.110.254:35959)
0926 06:17:32.348734 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 06:17:32.348737 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:17:32.348998 (+ 261us) server_negotiation.cc:407] Connection header received
0926 06:17:32.349622 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:17:32.349623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:17:32.349636 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:17:32.349707 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:17:32.350301 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:32.350816 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:32.351953 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:32.352163 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:32.352256 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:32.355290 (+ 3034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:17:32.355292 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:17:32.355292 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:17:32.355300 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:17:32.356772 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:32.357231 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:32.357232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:32.357232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:32.357293 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:32.357596 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:32.357597 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:32.357598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:32.357727 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:17:32.357804 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 06:17:32.358099 (+ 295us) server_negotiation.cc:299] Negotiation successful
0926 06:17:32.358194 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 06:17:32.358494 15207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:32.348679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35959 (local address 127.2.110.195:43533)
0926 06:17:32.348850 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:32.348892 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 06:17:32.349512 (+ 620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:17:32.349789 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:17:32.349792 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:17:32.349982 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:17:32.350190 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:32.350194 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:32.350921 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:32.350922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:32.351856 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:32.351858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:32.352278 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:32.352279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:32.352400 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:32.353693 (+ 1293us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:17:32.353705 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:17:32.355187 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:17:32.356881 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:17:32.356889 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:17:32.356892 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:17:32.357140 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:17:32.357377 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:17:32.357378 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:17:32.357378 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:17:32.357517 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:17:32.357796 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:17:32.357802 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:17:32.357957 (+ 155us) client_negotiation.cc:769] Sending connection context
0926 06:17:32.358067 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 06:17:32.358233 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":96,"threads_started":1}
I20260926 06:17:32.359018 15205 heartbeater.cc:347] Connected to a master server at 127.2.110.254:35959
I20260926 06:17:32.359151 15205 heartbeater.cc:464] Registering TS with master...
I20260926 06:17:32.359349 15205 heartbeater.cc:511] Master 127.2.110.254:35959 requested a full tablet report, sending...
I20260926 06:17:32.359756 14720 ts_manager.cc:194] Registered new tserver with Master: 22a2a137f0f34c4590acc11d8139de50 (127.2.110.195:42875)
I20260926 06:17:32.360158 14720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.195@KRBTEST.COM'} at 127.2.110.195:43533
I20260926 06:17:32.365161 2491 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 06:17:32.365825 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 14802
I20260926 06:17:32.371299 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 14939
I20260926 06:17:32.377158 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 15075
I20260926 06:17:32.383332 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 14688
I20260926 06:17:32.391121 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:35959
--webserver_interface=127.2.110.254
--webserver_port=45077
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:41123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.110.254:35959
--ranger_config_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.110.212:41919/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.110.212:41919/kms with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:17:32.491048 15208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:17:32.491271 15208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:17:32.491294 15208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:17:32.492497 15208 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:17:32.492544 15208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:17:32.492559 15208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:17:32.492573 15208 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:17:32.492584 15208 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:17:32.494258 15208 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:41123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.110.212:41919/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.110.254:35959
--ranger_config_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:35959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.110.254
--webserver_port=45077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:17:32.494515 15208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:17:32.494774 15208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 06:17:32.499830 15208 server_base.cc:1034] running on GCE node
W20260926 06:17:32.499904 15216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:17:32.499799 15213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:17:32.499794 15214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:17:32.500267 15208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:17:32.500622 15208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:17:32.501791 15208 hybrid_clock.cc:648] HybridClock initialized: now 1790403452501778 us; error 30 us; skew 500 ppm
Sep 26 06:17:32 dist-test-slave-hndd krb5kdc[14244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403452, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:17:32.505213 15208 init.cc:377] Logged in from keytab as kudu/127.2.110.254@KRBTEST.COM (short username kudu)
I20260926 06:17:32.505815 15208 webserver.cc:460] Webserver started at http://127.2.110.254:45077/ using document root <none> and password file <none>
I20260926 06:17:32.506079 15208 fs_manager.cc:359] Metadata directory not provided
I20260926 06:17:32.506160 15208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 06:17:32 dist-test-slave-hndd krb5kdc[14244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403452, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.254@KRBTEST.COM for HTTP/127.2.110.212@KRBTEST.COM
I20260926 06:17:32.570425 15208 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:17:32.571317 15224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:17:32.571563 15208 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:17:32.571668 15208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "eb2e6df8b9ce4393bbb8e7f8dd777b37"
format_stamp: "Formatted at 2026-09-26 06:17:30 on dist-test-slave-hndd"
server_key: "ec7518fb7ddb0eada7dd0c566c196cb3"
server_key_iv: "b053c1fa670fb2c15e914871140a90a5"
server_key_version: "kuduclusterkey@0"
I20260926 06:17:32.572160 15208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:17:32.600919 15208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:17:32.603184 15208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:17:32.603312 15208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:17:32.608366 15208 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.254:35959
I20260926 06:17:32.608417 15276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.254:35959 every 8 connection(s)
I20260926 06:17:32.608824 15208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:17:32.609499 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 15208
I20260926 06:17:32.609998 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.110.212:41919/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.193:40231
--local_ip_for_outbound_sockets=127.2.110.193
--tserver_master_addrs=127.2.110.254:35959
--webserver_port=41039
--webserver_interface=127.2.110.193
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab
--principal=kudu/127.2.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:41123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.110.212:41919/kms with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 06:17:32.611204 15277 sys_catalog.cc:263] Verifying existing consensus state
I20260926 06:17:32.611996 15277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb2e6df8b9ce4393bbb8e7f8dd777b37: Bootstrap starting.
I20260926 06:17:32.620193 15277 log.cc:824] T 00000000000000000000000000000000 P eb2e6df8b9ce4393bbb8e7f8dd777b37: Log is configured to *not* fsync() on all Append() calls
I20260926 06:17:32.622125 15277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb2e6df8b9ce4393bbb8e7f8dd777b37: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 06:17:32.622442 15277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb2e6df8b9ce4393bbb8e7f8dd777b37: Bootstrap complete.
I20260926 06:17:32.624967 15277 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb2e6df8b9ce4393bbb8e7f8dd777b37 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb2e6df8b9ce4393bbb8e7f8dd777b37" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 35959 } }
I20260926 06:17:32.625216 15277 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb2e6df8b9ce4393bbb8e7f8dd777b37 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2e6df8b9ce4393bbb8e7f8dd777b37, State: Initialized, Role: FOLLOWER
I20260926 06:17:32.625411 15277 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb2e6df8b9ce4393bbb8e7f8dd777b37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb2e6df8b9ce4393bbb8e7f8dd777b37" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 35959 } }
I20260926 06:17:32.625492 15277 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb2e6df8b9ce4393bbb8e7f8dd777b37 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:17:32.625545 15277 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb2e6df8b9ce4393bbb8e7f8dd777b37 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:17:32.625604 15277 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb2e6df8b9ce4393bbb8e7f8dd777b37 [term 1 FOLLOWER]: Advancing to term 2
I20260926 06:17:32.626708 15277 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb2e6df8b9ce4393bbb8e7f8dd777b37 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb2e6df8b9ce4393bbb8e7f8dd777b37" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 35959 } }
I20260926 06:17:32.626803 15277 leader_election.cc:302] T 00000000000000000000000000000000 P eb2e6df8b9ce4393bbb8e7f8dd777b37 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb2e6df8b9ce4393bbb8e7f8dd777b37; no voters:
I20260926 06:17:32.627032 15277 leader_election.cc:288] T 00000000000000000000000000000000 P eb2e6df8b9ce4393bbb8e7f8dd777b37 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 06:17:32.627077 15281 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb2e6df8b9ce4393bbb8e7f8dd777b37 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 06:17:32.627225 15281 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb2e6df8b9ce4393bbb8e7f8dd777b37 [term 2 LEADER]: Becoming Leader. State: Replica: eb2e6df8b9ce4393bbb8e7f8dd777b37, State: Running, Role: LEADER
I20260926 06:17:32.627323 15277 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb2e6df8b9ce4393bbb8e7f8dd777b37 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:17:32.627409 15281 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb2e6df8b9ce4393bbb8e7f8dd777b37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb2e6df8b9ce4393bbb8e7f8dd777b37" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 35959 } }
I20260926 06:17:32.627879 15283 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb2e6df8b9ce4393bbb8e7f8dd777b37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb2e6df8b9ce4393bbb8e7f8dd777b37. Latest consensus state: current_term: 2 leader_uuid: "eb2e6df8b9ce4393bbb8e7f8dd777b37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb2e6df8b9ce4393bbb8e7f8dd777b37" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 35959 } } }
I20260926 06:17:32.627892 15282 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb2e6df8b9ce4393bbb8e7f8dd777b37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "eb2e6df8b9ce4393bbb8e7f8dd777b37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb2e6df8b9ce4393bbb8e7f8dd777b37" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 35959 } } }
I20260926 06:17:32.627978 15283 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb2e6df8b9ce4393bbb8e7f8dd777b37 [sys.catalog]: This master's current role is: LEADER
I20260926 06:17:32.628000 15282 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb2e6df8b9ce4393bbb8e7f8dd777b37 [sys.catalog]: This master's current role is: LEADER
I20260926 06:17:32.628295 15288 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:17:32.628861 15288 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:17:32.628999 15288 catalog_manager.cc:1259] Loaded cluster ID: 38198e1edd474deb8321df7dd1f2ab09
I20260926 06:17:32.629097 15288 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:17:32.629315 15277 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 06:17:32.630321 15288 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:17:32.630501 15288 catalog_manager.cc:5879] T 00000000000000000000000000000000 P eb2e6df8b9ce4393bbb8e7f8dd777b37: Loaded TSK: 0
I20260926 06:17:32.630714 15288 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 06:17:32.710361 15279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:17:32.710596 15279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:17:32.710640 15279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:17:32.711866 15279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:17:32.711946 15279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:17:32.711994 15279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.193
I20260926 06:17:32.713725 15279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:41123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.110.212:41919/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab
--principal=kudu/127.2.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.193:40231
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.110.193
--webserver_port=41039
--tserver_master_addrs=127.2.110.254:35959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.193
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:17:32.713997 15279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:17:32.714321 15279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:17:32.720897 15313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:17:32.721159 15279 server_base.cc:1034] running on GCE node
W20260926 06:17:32.720906 15312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:17:32.721819 15315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:17:32.722078 15279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:17:32.722438 15279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:17:32.723666 15279 hybrid_clock.cc:648] HybridClock initialized: now 1790403452723631 us; error 56 us; skew 500 ppm
Sep 26 06:17:32 dist-test-slave-hndd krb5kdc[14244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403452, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:17:32.727516 15279 init.cc:377] Logged in from keytab as kudu/127.2.110.193@KRBTEST.COM (short username kudu)
I20260926 06:17:32.728072 15279 webserver.cc:460] Webserver started at http://127.2.110.193:41039/ using document root <none> and password file <none>
I20260926 06:17:32.728353 15279 fs_manager.cc:359] Metadata directory not provided
I20260926 06:17:32.728500 15279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 06:17:32 dist-test-slave-hndd krb5kdc[14244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403452, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.193@KRBTEST.COM for HTTP/127.2.110.212@KRBTEST.COM
I20260926 06:17:32.759346 15279 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260926 06:17:32.760697 15325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:17:32.760926 15279 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:17:32.761027 15279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-0/wal
uuid: "61a97084ca5945f889a2aa4893685ed5"
format_stamp: "Formatted at 2026-09-26 06:17:31 on dist-test-slave-hndd"
server_key: "c832b9aa050945aaee53f182168acae5"
server_key_iv: "57aa8078a6ff3974262f308ff3957ad1"
server_key_version: "kuduclusterkey@0"
I20260926 06:17:32.761566 15279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:17:32.779167 15279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:17:32.781733 15279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:17:32.781961 15279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:17:32.782274 15279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:17:32.782677 15279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:17:32.782771 15279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:17:32.782835 15279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:17:32.782871 15279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:17:32.789989 15279 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.193:40231
I20260926 06:17:32.790457 15279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
I20260926 06:17:32.793795 15438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.193:40231 every 8 connection(s)
I20260926 06:17:32.798821 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 15279
I20260926 06:17:32.799140 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.110.212:41919/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.194:43281
--local_ip_for_outbound_sockets=127.2.110.194
--tserver_master_addrs=127.2.110.254:35959
--webserver_port=37527
--webserver_interface=127.2.110.194
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab
--principal=kudu/127.2.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:41123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.110.212:41919/kms with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 06:17:32 dist-test-slave-hndd krb5kdc[14244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403452, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.193@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:17:32.810750 15441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:32.791224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35959 (local address 127.2.110.193:42101)
0926 06:17:32.797448 (+ 6224us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:32.797510 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 06:17:32.798224 (+ 714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:17:32.798955 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:17:32.798960 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:17:32.799222 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:17:32.799551 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:32.799558 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:32.801299 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:32.801300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:32.802643 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:32.802647 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:32.803657 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:32.803658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:32.803793 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:32.804449 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:17:32.804459 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:17:32.806274 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:17:32.808751 (+ 2477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:17:32.808759 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:17:32.808762 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:17:32.809071 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:17:32.809404 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:17:32.809405 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:17:32.809405 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:17:32.809549 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:17:32.810062 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:17:32.810070 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:17:32.810225 (+ 155us) client_negotiation.cc:769] Sending connection context
0926 06:17:32.810335 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 06:17:32.810502 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6165,"thread_start_us":76,"threads_started":1}
I20260926 06:17:32.810792 15442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:32.791306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:42101 (local address 127.2.110.254:35959)
0926 06:17:32.791522 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 06:17:32.791531 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 06:17:32.797680 (+ 6149us) server_negotiation.cc:407] Connection header received
0926 06:17:32.798500 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:17:32.798508 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:17:32.798572 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:17:32.798726 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:17:32.799691 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:32.801171 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:32.802746 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:32.803425 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:32.803535 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:32.806431 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:17:32.806437 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:17:32.806437 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:17:32.806450 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:17:32.808646 (+ 2196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:32.809186 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:32.809190 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:32.809190 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:32.809298 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:32.809630 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:32.809631 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:32.809631 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:32.809941 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:17:32.810080 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 06:17:32.810444 (+ 364us) server_negotiation.cc:299] Negotiation successful
0926 06:17:32.810586 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":84,"threads_started":1}
I20260926 06:17:32.812261 15439 heartbeater.cc:347] Connected to a master server at 127.2.110.254:35959
I20260926 06:17:32.812369 15439 heartbeater.cc:464] Registering TS with master...
I20260926 06:17:32.812597 15439 heartbeater.cc:511] Master 127.2.110.254:35959 requested a full tablet report, sending...
W20260926 06:17:32.813642 15439 heartbeater.cc:644] Failed to heartbeat to 127.2.110.254:35959 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 06:17:32.813719 15439 heartbeater.cc:464] Registering TS with master...
I20260926 06:17:32.813786 15439 heartbeater.cc:511] Master 127.2.110.254:35959 requested a full tablet report, sending...
I20260926 06:17:32.814255 15439 heartbeater.cc:464] Registering TS with master...
I20260926 06:17:32.814337 15439 heartbeater.cc:511] Master 127.2.110.254:35959 requested a full tablet report, sending...
W20260926 06:17:32.814685 15439 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 06:17:32.916553 15443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:17:32.916949 15443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:17:32.917032 15443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:17:32.920758 15443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:17:32.921023 15443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:17:32.921139 15443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.194
I20260926 06:17:32.924074 15443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:41123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.110.212:41919/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab
--principal=kudu/127.2.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.194:43281
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.110.194
--webserver_port=37527
--tserver_master_addrs=127.2.110.254:35959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.194
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:17:32.924413 15443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:17:32.924707 15443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:17:32.930759 15448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:17:32.930759 15449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:17:32.931046 15451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:17:32.932199 15443 server_base.cc:1034] running on GCE node
I20260926 06:17:32.932420 15443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:17:32.932704 15443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:17:32.933869 15443 hybrid_clock.cc:648] HybridClock initialized: now 1790403452933835 us; error 46 us; skew 500 ppm
Sep 26 06:17:32 dist-test-slave-hndd krb5kdc[14244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403452, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:17:32.937204 15443 init.cc:377] Logged in from keytab as kudu/127.2.110.194@KRBTEST.COM (short username kudu)
I20260926 06:17:32.937796 15443 webserver.cc:460] Webserver started at http://127.2.110.194:37527/ using document root <none> and password file <none>
I20260926 06:17:32.938136 15443 fs_manager.cc:359] Metadata directory not provided
I20260926 06:17:32.938230 15443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 06:17:32 dist-test-slave-hndd krb5kdc[14244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403452, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.194@KRBTEST.COM for HTTP/127.2.110.212@KRBTEST.COM
I20260926 06:17:32.963178 15443 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260926 06:17:32.964291 15459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:17:32.964520 15443 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:17:32.964653 15443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-1/wal
uuid: "5131ae7792e44f7f974f659ae02595da"
format_stamp: "Formatted at 2026-09-26 06:17:31 on dist-test-slave-hndd"
server_key: "36ec4ede4483576d6cb2423d7dceef88"
server_key_iv: "7a442223b181d1b346a965f3e2779909"
server_key_version: "kuduclusterkey@0"
I20260926 06:17:32.965075 15443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:17:32.975468 15443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:17:32.978470 15443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:17:32.978719 15443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:17:32.979055 15443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:17:32.979487 15443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:17:32.979578 15443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:17:32.979641 15443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:17:32.979689 15443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:17:32.987757 15443 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.194:43281
I20260926 06:17:32.988243 15443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
I20260926 06:17:32.988854 15572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.194:43281 every 8 connection(s)
I20260926 06:17:32.994525 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 15443
I20260926 06:17:32.994869 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.110.212:41919/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.195:42875
--local_ip_for_outbound_sockets=127.2.110.195
--tserver_master_addrs=127.2.110.254:35959
--webserver_port=33259
--webserver_interface=127.2.110.195
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab
--principal=kudu/127.2.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:41123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.110.212:41919/kms with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 06:17:32 dist-test-slave-hndd krb5kdc[14244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403452, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.194@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:17:33.002140 15442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:32.989413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:39975 (local address 127.2.110.254:35959)
0926 06:17:32.989459 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 06:17:32.989461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:17:32.989474 (+ 13us) server_negotiation.cc:407] Connection header received
0926 06:17:32.990084 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:17:32.990085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:17:32.990098 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:17:32.990167 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:17:32.990856 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:32.991617 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:32.993162 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:32.993531 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:32.993610 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:32.998353 (+ 4743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:17:32.998355 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:17:32.998356 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:17:32.998366 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:17:33.000214 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:33.000854 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:33.000856 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:33.000856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:33.000928 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:33.001276 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:33.001277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:33.001277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:33.001458 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:17:33.001541 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 06:17:33.001941 (+ 400us) server_negotiation.cc:299] Negotiation successful
0926 06:17:33.002040 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 06:17:33.002398 15575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:32.989038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35959 (local address 127.2.110.194:39975)
0926 06:17:32.989236 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:32.989285 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 06:17:32.989956 (+ 671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:17:32.990236 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:17:32.990240 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:17:32.990474 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:17:32.990751 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:32.990757 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:32.991749 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:32.991750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:32.993032 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:32.993035 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:32.993718 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:32.993720 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:32.993874 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:32.996556 (+ 2682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:17:32.996568 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:17:32.998189 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:17:33.000342 (+ 2153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:17:33.000353 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:17:33.000356 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:17:33.000746 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:17:33.001017 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:17:33.001018 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:17:33.001019 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:17:33.001187 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:17:33.001543 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:17:33.001550 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:17:33.001758 (+ 208us) client_negotiation.cc:769] Sending connection context
0926 06:17:33.001896 (+ 138us) client_negotiation.cc:240] Negotiation successful
0926 06:17:33.002093 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":92,"threads_started":1}
I20260926 06:17:33.004196 15573 heartbeater.cc:347] Connected to a master server at 127.2.110.254:35959
I20260926 06:17:33.004371 15573 heartbeater.cc:464] Registering TS with master...
I20260926 06:17:33.004657 15573 heartbeater.cc:511] Master 127.2.110.254:35959 requested a full tablet report, sending...
W20260926 06:17:33.005393 15573 heartbeater.cc:644] Failed to heartbeat to 127.2.110.254:35959 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 06:17:33.005529 15573 heartbeater.cc:464] Registering TS with master...
I20260926 06:17:33.005666 15573 heartbeater.cc:511] Master 127.2.110.254:35959 requested a full tablet report, sending...
I20260926 06:17:33.006037 15573 heartbeater.cc:464] Registering TS with master...
I20260926 06:17:33.006156 15573 heartbeater.cc:511] Master 127.2.110.254:35959 requested a full tablet report, sending...
W20260926 06:17:33.006500 15573 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 06:17:33.099977 15576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:17:33.100302 15576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:17:33.100378 15576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:17:33.102020 15576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:17:33.102113 15576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:17:33.102162 15576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.195
I20260926 06:17:33.103771 15576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:41123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.110.212:41919/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab
--principal=kudu/127.2.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.195:42875
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.110.195
--webserver_port=33259
--tserver_master_addrs=127.2.110.254:35959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.195
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:17:33.104071 15576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:17:33.104365 15576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:17:33.110522 15582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:17:33.110520 15581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:17:33.113559 15584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:17:33.115041 15576 server_base.cc:1034] running on GCE node
I20260926 06:17:33.115363 15576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:17:33.115700 15576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:17:33.116894 15576 hybrid_clock.cc:648] HybridClock initialized: now 1790403453116858 us; error 51 us; skew 500 ppm
Sep 26 06:17:33 dist-test-slave-hndd krb5kdc[14244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403453, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:17:33.121832 15576 init.cc:377] Logged in from keytab as kudu/127.2.110.195@KRBTEST.COM (short username kudu)
I20260926 06:17:33.122453 15576 webserver.cc:460] Webserver started at http://127.2.110.195:33259/ using document root <none> and password file <none>
I20260926 06:17:33.122821 15576 fs_manager.cc:359] Metadata directory not provided
I20260926 06:17:33.122975 15576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 06:17:33 dist-test-slave-hndd krb5kdc[14244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403453, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.195@KRBTEST.COM for HTTP/127.2.110.212@KRBTEST.COM
I20260926 06:17:33.139192 15576 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:17:33.139989 15593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:17:33.140156 15576 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:17:33.140240 15576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-2/wal
uuid: "22a2a137f0f34c4590acc11d8139de50"
format_stamp: "Formatted at 2026-09-26 06:17:32 on dist-test-slave-hndd"
server_key: "530eaa174fd207176d40a60e8ea40d1e"
server_key_iv: "cb2a536319c8cfbc21670e4ec9c2558b"
server_key_version: "kuduclusterkey@0"
I20260926 06:17:33.141404 15576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:17:33.156212 15576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:17:33.158890 15576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:17:33.159159 15576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:17:33.159464 15576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:17:33.159930 15576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:17:33.160041 15576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:17:33.160136 15576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:17:33.160207 15576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:17:33.167523 15576 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.195:42875
I20260926 06:17:33.167621 15706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.195:42875 every 8 connection(s)
I20260926 06:17:33.168023 15576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
I20260926 06:17:33.173278 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 15576
Sep 26 06:17:33 dist-test-slave-hndd krb5kdc[14244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403453, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.195@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:17:33.180948 15442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:33.168854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:58897 (local address 127.2.110.254:35959)
0926 06:17:33.168925 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 06:17:33.168927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:17:33.169187 (+ 260us) server_negotiation.cc:407] Connection header received
0926 06:17:33.169932 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:17:33.169934 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:17:33.169945 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:17:33.170074 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:17:33.170859 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:33.171376 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:33.172943 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:33.173416 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:33.173493 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:33.176993 (+ 3500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:17:33.176995 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:17:33.176995 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:17:33.177003 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:17:33.178886 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:33.179485 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:33.179486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:33.179487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:33.179556 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:33.179895 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:33.179896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:33.179897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:33.180043 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:17:33.180110 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 06:17:33.180761 (+ 651us) server_negotiation.cc:299] Negotiation successful
0926 06:17:33.180849 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 06:17:33.181638 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:33.168804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35959 (local address 127.2.110.195:58897)
0926 06:17:33.169002 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:33.169050 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 06:17:33.169806 (+ 756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:17:33.170148 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:17:33.170152 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:17:33.170439 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:17:33.170746 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:33.170752 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:33.171478 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:33.171479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:33.172774 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:33.172777 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:33.173595 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:33.173595 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:33.173698 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:33.174293 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:17:33.174302 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:17:33.176877 (+ 2575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:17:33.178998 (+ 2121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:17:33.179009 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:17:33.179012 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:17:33.179379 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:17:33.179642 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:17:33.179643 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:17:33.179643 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:17:33.179816 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:17:33.180158 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:17:33.180165 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:17:33.180351 (+ 186us) client_negotiation.cc:769] Sending connection context
0926 06:17:33.180473 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 06:17:33.180662 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":83,"threads_started":1}
I20260926 06:17:33.182683 15707 heartbeater.cc:347] Connected to a master server at 127.2.110.254:35959
I20260926 06:17:33.182817 15707 heartbeater.cc:464] Registering TS with master...
I20260926 06:17:33.183032 15707 heartbeater.cc:511] Master 127.2.110.254:35959 requested a full tablet report, sending...
W20260926 06:17:33.183740 15707 heartbeater.cc:644] Failed to heartbeat to 127.2.110.254:35959 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 06:17:33.183810 15707 heartbeater.cc:464] Registering TS with master...
I20260926 06:17:33.183874 15707 heartbeater.cc:511] Master 127.2.110.254:35959 requested a full tablet report, sending...
I20260926 06:17:33.184235 15707 heartbeater.cc:464] Registering TS with master...
I20260926 06:17:33.184312 15707 heartbeater.cc:511] Master 127.2.110.254:35959 requested a full tablet report, sending...
W20260926 06:17:33.184677 15707 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 06:17:33.185690 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:33.173913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42984 (local address 127.2.110.254:35959)
0926 06:17:33.174057 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 06:17:33.174059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:17:33.174140 (+ 81us) server_negotiation.cc:407] Connection header received
0926 06:17:33.174861 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:17:33.174862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:17:33.174873 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:17:33.174928 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:17:33.176575 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:33.177421 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:33.179916 (+ 2495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:33.180224 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:33.180294 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:33.181881 (+ 1587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:17:33.181883 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:17:33.181883 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:17:33.181890 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:17:33.183736 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:33.184773 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:33.184775 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:33.184775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:33.184845 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:33.185065 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:33.185066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:33.185067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:33.185218 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:17:33.185427 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0926 06:17:33.185471 (+ 44us) server_negotiation.cc:299] Negotiation successful
0926 06:17:33.185565 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":68,"threads_started":1}
Sep 26 06:17:33 dist-test-slave-hndd krb5kdc[14244](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790403453, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:17:33.814857 15439 heartbeater.cc:464] Registering TS with master...
I20260926 06:17:33.815016 15439 heartbeater.cc:511] Master 127.2.110.254:35959 requested a full tablet report, sending...
I20260926 06:17:33.815419 15241 ts_manager.cc:194] Registered new tserver with Master: 61a97084ca5945f889a2aa4893685ed5 (127.2.110.193:40231)
I20260926 06:17:33.816674 15241 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.193@KRBTEST.COM'} at 127.2.110.193:42101
I20260926 06:17:34.006762 15573 heartbeater.cc:464] Registering TS with master...
I20260926 06:17:34.006964 15573 heartbeater.cc:511] Master 127.2.110.254:35959 requested a full tablet report, sending...
I20260926 06:17:34.007352 15241 ts_manager.cc:194] Registered new tserver with Master: 5131ae7792e44f7f974f659ae02595da (127.2.110.194:43281)
I20260926 06:17:34.007936 15241 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.194@KRBTEST.COM'} at 127.2.110.194:39975
Sep 26 06:17:34 dist-test-slave-hndd krb5kdc[14244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403453, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.254@KRBTEST.COM for HTTP/127.2.110.212@KRBTEST.COM
I20260926 06:17:34.184867 15707 heartbeater.cc:464] Registering TS with master...
I20260926 06:17:34.185045 15707 heartbeater.cc:511] Master 127.2.110.254:35959 requested a full tablet report, sending...
I20260926 06:17:34.185406 15241 ts_manager.cc:194] Registered new tserver with Master: 22a2a137f0f34c4590acc11d8139de50 (127.2.110.195:42875)
I20260926 06:17:34.185909 15241 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.195@KRBTEST.COM'} at 127.2.110.195:58897
I20260926 06:17:34.192687 2491 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 06:17:34.206475 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:34.193813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42998 (local address 127.2.110.254:35959)
0926 06:17:34.194098 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0926 06:17:34.194100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:17:34.194263 (+ 163us) server_negotiation.cc:407] Connection header received
0926 06:17:34.194289 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:17:34.194289 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:17:34.194301 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:17:34.194357 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:17:34.195087 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:34.195918 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:34.197255 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:34.197591 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:34.197663 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:34.202879 (+ 5216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:17:34.202881 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:17:34.202881 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:17:34.202891 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:17:34.204812 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:34.205242 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:34.205243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:34.205244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:34.205303 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:34.205525 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:34.205526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:34.205526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:34.205674 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:17:34.205774 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 06:17:34.206269 (+ 495us) server_negotiation.cc:299] Negotiation successful
0926 06:17:34.206359 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":92,"threads_started":1}
I20260926 06:17:34.209069 15241 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42998:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:17:34.211367 15241 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:17:34.231658 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:34.224888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.195:42875 (local address 127.0.0.1:44544)
0926 06:17:34.226862 (+ 1974us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:34.226916 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 06:17:34.226991 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:17:34.227538 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:17:34.227541 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:17:34.227544 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:17:34.227620 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:34.227622 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:34.228914 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:34.228915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:34.230240 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:34.230243 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:34.231322 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:34.231323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:34.231408 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:34.231422 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 06:17:34.231493 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 06:17:34.231554 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1937,"thread_start_us":93,"threads_started":1}
I20260926 06:17:34.232652 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:34.225742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44544 (local address 127.2.110.195:42875)
0926 06:17:34.227144 (+ 1402us) server_negotiation.cc:206] Beginning negotiation
0926 06:17:34.227150 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 06:17:34.227161 (+ 11us) server_negotiation.cc:407] Connection header received
0926 06:17:34.227197 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:17:34.227199 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:17:34.227216 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:17:34.227287 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:17:34.227707 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:34.228814 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:34.230342 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:34.231138 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:34.231215 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:34.231281 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 06:17:34.232516 (+ 1235us) server_negotiation.cc:299] Negotiation successful
0926 06:17:34.232568 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1367,"thread_start_us":75,"threads_started":1}
I20260926 06:17:34.237021 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:34.225136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.194:43281 (local address 127.0.0.1:51096)
0926 06:17:34.229435 (+ 4299us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:34.229448 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:17:34.229509 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:17:34.233054 (+ 3545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:17:34.233055 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:17:34.233058 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:17:34.233094 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:34.233095 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:34.234502 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:34.234502 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:34.235592 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:34.235595 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:34.236740 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:34.236741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:34.236808 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:34.236813 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 06:17:34.236869 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 06:17:34.236922 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4281,"mutex_wait_us":29,"thread_start_us":64,"threads_started":1}
I20260926 06:17:34.237481 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:34.225898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51096 (local address 127.2.110.194:43281)
0926 06:17:34.227421 (+ 1523us) server_negotiation.cc:206] Beginning negotiation
0926 06:17:34.227425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:17:34.229579 (+ 2154us) server_negotiation.cc:407] Connection header received
0926 06:17:34.229619 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:17:34.229622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:17:34.229638 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:17:34.229707 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:17:34.233187 (+ 3480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:34.234362 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:34.235713 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:34.236534 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:34.236627 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:34.236697 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 06:17:34.237319 (+ 622us) server_negotiation.cc:299] Negotiation successful
0926 06:17:34.237400 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1492,"thread_start_us":49,"threads_started":1}
I20260926 06:17:34.238543 15507 tablet_service.cc:1467] Processing CreateTablet for tablet 7dfbaa255ace4c85844a78a67a38354c (DEFAULT_TABLE table=test-table [id=7ca51787286d4129943c8ad30487256e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:17:34.238693 15641 tablet_service.cc:1467] Processing CreateTablet for tablet 7dfbaa255ace4c85844a78a67a38354c (DEFAULT_TABLE table=test-table [id=7ca51787286d4129943c8ad30487256e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:17:34.238986 15507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dfbaa255ace4c85844a78a67a38354c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:17:34.239042 15641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dfbaa255ace4c85844a78a67a38354c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:17:34.240002 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:34.225275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.193:40231 (local address 127.0.0.1:39896)
0926 06:17:34.232142 (+ 6867us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:34.232153 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:17:34.232193 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:17:34.232899 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:17:34.232901 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:17:34.232903 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:17:34.232942 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:34.232943 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:34.234427 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:34.234428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:34.238623 (+ 4195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:34.238627 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:34.239754 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:34.239755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:34.239821 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:34.239825 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:17:34.239873 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 06:17:34.239926 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6854,"mutex_wait_us":3,"thread_start_us":51,"threads_started":1}
I20260926 06:17:34.240275 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:34.226025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39896 (local address 127.2.110.193:40231)
0926 06:17:34.227495 (+ 1470us) server_negotiation.cc:206] Beginning negotiation
0926 06:17:34.227499 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:17:34.232262 (+ 4763us) server_negotiation.cc:407] Connection header received
0926 06:17:34.232307 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:17:34.232311 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:17:34.232329 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:17:34.232408 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:17:34.233035 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:34.234313 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:34.238737 (+ 4424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:34.239532 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:34.239617 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:34.239678 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 06:17:34.240154 (+ 476us) server_negotiation.cc:299] Negotiation successful
0926 06:17:34.240201 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1434,"thread_start_us":62,"threads_started":1}
I20260926 06:17:34.240929 15373 tablet_service.cc:1467] Processing CreateTablet for tablet 7dfbaa255ace4c85844a78a67a38354c (DEFAULT_TABLE table=test-table [id=7ca51787286d4129943c8ad30487256e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:17:34.241281 15373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dfbaa255ace4c85844a78a67a38354c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:17:34.245417 15742 tablet_bootstrap.cc:492] T 7dfbaa255ace4c85844a78a67a38354c P 61a97084ca5945f889a2aa4893685ed5: Bootstrap starting.
I20260926 06:17:34.246281 15742 tablet_bootstrap.cc:654] T 7dfbaa255ace4c85844a78a67a38354c P 61a97084ca5945f889a2aa4893685ed5: Neither blocks nor log segments found. Creating new log.
I20260926 06:17:34.246729 15742 log.cc:824] T 7dfbaa255ace4c85844a78a67a38354c P 61a97084ca5945f889a2aa4893685ed5: Log is configured to *not* fsync() on all Append() calls
I20260926 06:17:34.247169 15740 tablet_bootstrap.cc:492] T 7dfbaa255ace4c85844a78a67a38354c P 22a2a137f0f34c4590acc11d8139de50: Bootstrap starting.
I20260926 06:17:34.247531 15741 tablet_bootstrap.cc:492] T 7dfbaa255ace4c85844a78a67a38354c P 5131ae7792e44f7f974f659ae02595da: Bootstrap starting.
I20260926 06:17:34.248330 15740 tablet_bootstrap.cc:654] T 7dfbaa255ace4c85844a78a67a38354c P 22a2a137f0f34c4590acc11d8139de50: Neither blocks nor log segments found. Creating new log.
I20260926 06:17:34.249017 15741 tablet_bootstrap.cc:654] T 7dfbaa255ace4c85844a78a67a38354c P 5131ae7792e44f7f974f659ae02595da: Neither blocks nor log segments found. Creating new log.
I20260926 06:17:34.249441 15741 log.cc:824] T 7dfbaa255ace4c85844a78a67a38354c P 5131ae7792e44f7f974f659ae02595da: Log is configured to *not* fsync() on all Append() calls
I20260926 06:17:34.249841 15740 log.cc:824] T 7dfbaa255ace4c85844a78a67a38354c P 22a2a137f0f34c4590acc11d8139de50: Log is configured to *not* fsync() on all Append() calls
I20260926 06:17:34.250985 15742 tablet_bootstrap.cc:492] T 7dfbaa255ace4c85844a78a67a38354c P 61a97084ca5945f889a2aa4893685ed5: No bootstrap required, opened a new log
I20260926 06:17:34.251117 15742 ts_tablet_manager.cc:1397] T 7dfbaa255ace4c85844a78a67a38354c P 61a97084ca5945f889a2aa4893685ed5: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260926 06:17:34.253572 15742 raft_consensus.cc:348] T 7dfbaa255ace4c85844a78a67a38354c P 61a97084ca5945f889a2aa4893685ed5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a2a137f0f34c4590acc11d8139de50" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 42875 } } peers { permanent_uuid: "5131ae7792e44f7f974f659ae02595da" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 43281 } } peers { permanent_uuid: "61a97084ca5945f889a2aa4893685ed5" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 40231 } }
I20260926 06:17:34.253933 15742 raft_consensus.cc:374] T 7dfbaa255ace4c85844a78a67a38354c P 61a97084ca5945f889a2aa4893685ed5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:17:34.254034 15742 raft_consensus.cc:729] T 7dfbaa255ace4c85844a78a67a38354c P 61a97084ca5945f889a2aa4893685ed5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61a97084ca5945f889a2aa4893685ed5, State: Initialized, Role: FOLLOWER
I20260926 06:17:34.254277 15742 consensus_queue.cc:260] T 7dfbaa255ace4c85844a78a67a38354c P 61a97084ca5945f889a2aa4893685ed5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a2a137f0f34c4590acc11d8139de50" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 42875 } } peers { permanent_uuid: "5131ae7792e44f7f974f659ae02595da" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 43281 } } peers { permanent_uuid: "61a97084ca5945f889a2aa4893685ed5" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 40231 } }
I20260926 06:17:34.254684 15742 ts_tablet_manager.cc:1428] T 7dfbaa255ace4c85844a78a67a38354c P 61a97084ca5945f889a2aa4893685ed5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 06:17:34.254860 15439 heartbeater.cc:503] Master 127.2.110.254:35959 was elected leader, sending a full tablet report...
I20260926 06:17:34.256942 15741 tablet_bootstrap.cc:492] T 7dfbaa255ace4c85844a78a67a38354c P 5131ae7792e44f7f974f659ae02595da: No bootstrap required, opened a new log
I20260926 06:17:34.257081 15741 ts_tablet_manager.cc:1397] T 7dfbaa255ace4c85844a78a67a38354c P 5131ae7792e44f7f974f659ae02595da: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.002s
I20260926 06:17:34.259784 15741 raft_consensus.cc:348] T 7dfbaa255ace4c85844a78a67a38354c P 5131ae7792e44f7f974f659ae02595da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a2a137f0f34c4590acc11d8139de50" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 42875 } } peers { permanent_uuid: "5131ae7792e44f7f974f659ae02595da" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 43281 } } peers { permanent_uuid: "61a97084ca5945f889a2aa4893685ed5" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 40231 } }
I20260926 06:17:34.260056 15741 raft_consensus.cc:374] T 7dfbaa255ace4c85844a78a67a38354c P 5131ae7792e44f7f974f659ae02595da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:17:34.260154 15741 raft_consensus.cc:729] T 7dfbaa255ace4c85844a78a67a38354c P 5131ae7792e44f7f974f659ae02595da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5131ae7792e44f7f974f659ae02595da, State: Initialized, Role: FOLLOWER
I20260926 06:17:34.260375 15741 consensus_queue.cc:260] T 7dfbaa255ace4c85844a78a67a38354c P 5131ae7792e44f7f974f659ae02595da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a2a137f0f34c4590acc11d8139de50" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 42875 } } peers { permanent_uuid: "5131ae7792e44f7f974f659ae02595da" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 43281 } } peers { permanent_uuid: "61a97084ca5945f889a2aa4893685ed5" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 40231 } }
I20260926 06:17:34.260768 15741 ts_tablet_manager.cc:1428] T 7dfbaa255ace4c85844a78a67a38354c P 5131ae7792e44f7f974f659ae02595da: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260926 06:17:34.260946 15573 heartbeater.cc:503] Master 127.2.110.254:35959 was elected leader, sending a full tablet report...
I20260926 06:17:34.262217 15740 tablet_bootstrap.cc:492] T 7dfbaa255ace4c85844a78a67a38354c P 22a2a137f0f34c4590acc11d8139de50: No bootstrap required, opened a new log
I20260926 06:17:34.262360 15740 ts_tablet_manager.cc:1397] T 7dfbaa255ace4c85844a78a67a38354c P 22a2a137f0f34c4590acc11d8139de50: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20260926 06:17:34.265120 15740 raft_consensus.cc:348] T 7dfbaa255ace4c85844a78a67a38354c P 22a2a137f0f34c4590acc11d8139de50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a2a137f0f34c4590acc11d8139de50" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 42875 } } peers { permanent_uuid: "5131ae7792e44f7f974f659ae02595da" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 43281 } } peers { permanent_uuid: "61a97084ca5945f889a2aa4893685ed5" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 40231 } }
I20260926 06:17:34.265466 15740 raft_consensus.cc:374] T 7dfbaa255ace4c85844a78a67a38354c P 22a2a137f0f34c4590acc11d8139de50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:17:34.265568 15740 raft_consensus.cc:729] T 7dfbaa255ace4c85844a78a67a38354c P 22a2a137f0f34c4590acc11d8139de50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22a2a137f0f34c4590acc11d8139de50, State: Initialized, Role: FOLLOWER
I20260926 06:17:34.265794 15740 consensus_queue.cc:260] T 7dfbaa255ace4c85844a78a67a38354c P 22a2a137f0f34c4590acc11d8139de50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a2a137f0f34c4590acc11d8139de50" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 42875 } } peers { permanent_uuid: "5131ae7792e44f7f974f659ae02595da" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 43281 } } peers { permanent_uuid: "61a97084ca5945f889a2aa4893685ed5" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 40231 } }
I20260926 06:17:34.266309 15740 ts_tablet_manager.cc:1428] T 7dfbaa255ace4c85844a78a67a38354c P 22a2a137f0f34c4590acc11d8139de50: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260926 06:17:34.266518 15707 heartbeater.cc:503] Master 127.2.110.254:35959 was elected leader, sending a full tablet report...
W20260926 06:17:34.298611 15440 tablet.cc:2369] T 7dfbaa255ace4c85844a78a67a38354c P 61a97084ca5945f889a2aa4893685ed5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:17:34.332101 15748 raft_consensus.cc:482] T 7dfbaa255ace4c85844a78a67a38354c P 22a2a137f0f34c4590acc11d8139de50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:17:34.332265 15748 raft_consensus.cc:504] T 7dfbaa255ace4c85844a78a67a38354c P 22a2a137f0f34c4590acc11d8139de50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a2a137f0f34c4590acc11d8139de50" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 42875 } } peers { permanent_uuid: "5131ae7792e44f7f974f659ae02595da" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 43281 } } peers { permanent_uuid: "61a97084ca5945f889a2aa4893685ed5" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 40231 } }
I20260926 06:17:34.332670 15748 leader_election.cc:288] T 7dfbaa255ace4c85844a78a67a38354c P 22a2a137f0f34c4590acc11d8139de50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5131ae7792e44f7f974f659ae02595da (127.2.110.194:43281), 61a97084ca5945f889a2aa4893685ed5 (127.2.110.193:40231)
I20260926 06:17:34.336903 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:34.332964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:51311 (local address 127.2.110.194:43281)
0926 06:17:34.333013 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 06:17:34.333015 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:17:34.333217 (+ 202us) server_negotiation.cc:407] Connection header received
0926 06:17:34.333323 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:17:34.333324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:17:34.333354 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:17:34.333384 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:17:34.333633 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:34.334372 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:34.335896 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:34.336467 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:34.336516 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:34.336526 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 06:17:34.336778 (+ 252us) server_negotiation.cc:299] Negotiation successful
0926 06:17:34.336822 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 06:17:34.337433 15749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:34.332867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.194:43281 (local address 127.2.110.195:51311)
0926 06:17:34.333134 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:34.333146 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:17:34.333247 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:17:34.333457 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:17:34.333458 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:17:34.333460 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:17:34.333545 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:34.333547 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:34.334484 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:34.334485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:34.335770 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:34.335772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:34.336615 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:34.336616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:34.336694 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:34.336698 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:17:34.337242 (+ 544us) client_negotiation.cc:240] Negotiation successful
0926 06:17:34.337296 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":107,"threads_started":1}
I20260926 06:17:34.337978 15527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfbaa255ace4c85844a78a67a38354c" candidate_uuid: "22a2a137f0f34c4590acc11d8139de50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5131ae7792e44f7f974f659ae02595da" is_pre_election: true
I20260926 06:17:34.338166 15527 raft_consensus.cc:2454] T 7dfbaa255ace4c85844a78a67a38354c P 5131ae7792e44f7f974f659ae02595da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22a2a137f0f34c4590acc11d8139de50 in term 0.
I20260926 06:17:34.338428 15596 leader_election.cc:302] T 7dfbaa255ace4c85844a78a67a38354c P 22a2a137f0f34c4590acc11d8139de50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22a2a137f0f34c4590acc11d8139de50, 5131ae7792e44f7f974f659ae02595da; no voters:
I20260926 06:17:34.338586 15748 raft_consensus.cc:2781] T 7dfbaa255ace4c85844a78a67a38354c P 22a2a137f0f34c4590acc11d8139de50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:17:34.338637 15748 raft_consensus.cc:482] T 7dfbaa255ace4c85844a78a67a38354c P 22a2a137f0f34c4590acc11d8139de50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:17:34.338660 15748 raft_consensus.cc:3037] T 7dfbaa255ace4c85844a78a67a38354c P 22a2a137f0f34c4590acc11d8139de50 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:17:34.339648 15748 raft_consensus.cc:504] T 7dfbaa255ace4c85844a78a67a38354c P 22a2a137f0f34c4590acc11d8139de50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a2a137f0f34c4590acc11d8139de50" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 42875 } } peers { permanent_uuid: "5131ae7792e44f7f974f659ae02595da" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 43281 } } peers { permanent_uuid: "61a97084ca5945f889a2aa4893685ed5" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 40231 } }
I20260926 06:17:34.339802 15748 leader_election.cc:288] T 7dfbaa255ace4c85844a78a67a38354c P 22a2a137f0f34c4590acc11d8139de50 [CANDIDATE]: Term 1 election: Requested vote from peers 5131ae7792e44f7f974f659ae02595da (127.2.110.194:43281), 61a97084ca5945f889a2aa4893685ed5 (127.2.110.193:40231)
I20260926 06:17:34.340004 15527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfbaa255ace4c85844a78a67a38354c" candidate_uuid: "22a2a137f0f34c4590acc11d8139de50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5131ae7792e44f7f974f659ae02595da"
I20260926 06:17:34.340090 15527 raft_consensus.cc:3037] T 7dfbaa255ace4c85844a78a67a38354c P 5131ae7792e44f7f974f659ae02595da [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:17:34.340991 15527 raft_consensus.cc:2454] T 7dfbaa255ace4c85844a78a67a38354c P 5131ae7792e44f7f974f659ae02595da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22a2a137f0f34c4590acc11d8139de50 in term 1.
I20260926 06:17:34.341173 15596 leader_election.cc:302] T 7dfbaa255ace4c85844a78a67a38354c P 22a2a137f0f34c4590acc11d8139de50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22a2a137f0f34c4590acc11d8139de50, 5131ae7792e44f7f974f659ae02595da; no voters:
I20260926 06:17:34.341249 15748 raft_consensus.cc:2781] T 7dfbaa255ace4c85844a78a67a38354c P 22a2a137f0f34c4590acc11d8139de50 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:17:34.341410 15748 raft_consensus.cc:686] T 7dfbaa255ace4c85844a78a67a38354c P 22a2a137f0f34c4590acc11d8139de50 [term 1 LEADER]: Becoming Leader. State: Replica: 22a2a137f0f34c4590acc11d8139de50, State: Running, Role: LEADER
I20260926 06:17:34.341514 15748 consensus_queue.cc:237] T 7dfbaa255ace4c85844a78a67a38354c P 22a2a137f0f34c4590acc11d8139de50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a2a137f0f34c4590acc11d8139de50" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 42875 } } peers { permanent_uuid: "5131ae7792e44f7f974f659ae02595da" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 43281 } } peers { permanent_uuid: "61a97084ca5945f889a2aa4893685ed5" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 40231 } }
I20260926 06:17:34.341939 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:34.333284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:51093 (local address 127.2.110.193:40231)
0926 06:17:34.333326 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 06:17:34.333329 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:17:34.337786 (+ 4457us) server_negotiation.cc:407] Connection header received
0926 06:17:34.337945 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:17:34.337947 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:17:34.337960 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:17:34.338016 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:17:34.338356 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:34.339084 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:34.340753 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:34.341369 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:34.341418 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:34.341427 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 06:17:34.341822 (+ 395us) server_negotiation.cc:299] Negotiation successful
0926 06:17:34.341871 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 06:17:34.342278 15750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:34.333081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.193:40231 (local address 127.2.110.195:51093)
0926 06:17:34.337701 (+ 4620us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:34.337715 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 06:17:34.337820 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:17:34.338066 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:17:34.338066 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:17:34.338069 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:17:34.338270 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:34.338272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:34.339289 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:34.339290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:34.340647 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:34.340650 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:34.341584 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:34.341585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:34.341734 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:34.341738 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:17:34.342150 (+ 412us) client_negotiation.cc:240] Negotiation successful
0926 06:17:34.342197 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4600,"thread_start_us":61,"threads_started":1}
I20260926 06:17:34.342420 15241 catalog_manager.cc:5495] T 7dfbaa255ace4c85844a78a67a38354c P 22a2a137f0f34c4590acc11d8139de50 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22a2a137f0f34c4590acc11d8139de50 (127.2.110.195). New cstate: current_term: 1 leader_uuid: "22a2a137f0f34c4590acc11d8139de50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a2a137f0f34c4590acc11d8139de50" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 42875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5131ae7792e44f7f974f659ae02595da" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 43281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61a97084ca5945f889a2aa4893685ed5" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 40231 } health_report { overall_health: UNKNOWN } } }
I20260926 06:17:34.343358 15393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfbaa255ace4c85844a78a67a38354c" candidate_uuid: "22a2a137f0f34c4590acc11d8139de50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61a97084ca5945f889a2aa4893685ed5" is_pre_election: true
I20260926 06:17:34.343518 15393 raft_consensus.cc:2454] T 7dfbaa255ace4c85844a78a67a38354c P 61a97084ca5945f889a2aa4893685ed5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22a2a137f0f34c4590acc11d8139de50 in term 0.
I20260926 06:17:34.343639 15392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfbaa255ace4c85844a78a67a38354c" candidate_uuid: "22a2a137f0f34c4590acc11d8139de50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61a97084ca5945f889a2aa4893685ed5"
I20260926 06:17:34.343693 15392 raft_consensus.cc:3037] T 7dfbaa255ace4c85844a78a67a38354c P 61a97084ca5945f889a2aa4893685ed5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:17:34.344632 15392 raft_consensus.cc:2454] T 7dfbaa255ace4c85844a78a67a38354c P 61a97084ca5945f889a2aa4893685ed5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22a2a137f0f34c4590acc11d8139de50 in term 1.
I20260926 06:17:34.373778 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:34.370415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44556 (local address 127.2.110.195:42875)
0926 06:17:34.370459 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 06:17:34.370461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:17:34.370469 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:17:34.370511 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:17:34.370512 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:17:34.370534 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:17:34.370624 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:17:34.370793 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:34.371502 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:34.372579 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:34.372884 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:34.372967 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:34.373201 (+ 234us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:17:34.373370 (+ 169us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:17:34.373451 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 06:17:34.373672 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 06:17:34.373694 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 06:17:34.375980 15392 raft_consensus.cc:1261] T 7dfbaa255ace4c85844a78a67a38354c P 61a97084ca5945f889a2aa4893685ed5 [term 1 FOLLOWER]: Refusing update from remote peer 22a2a137f0f34c4590acc11d8139de50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:17:34.376309 15527 raft_consensus.cc:1261] T 7dfbaa255ace4c85844a78a67a38354c P 5131ae7792e44f7f974f659ae02595da [term 1 FOLLOWER]: Refusing update from remote peer 22a2a137f0f34c4590acc11d8139de50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:17:34.376354 15751 consensus_queue.cc:1035] T 7dfbaa255ace4c85844a78a67a38354c P 22a2a137f0f34c4590acc11d8139de50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61a97084ca5945f889a2aa4893685ed5" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 40231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:17:34.376667 15751 consensus_queue.cc:1035] T 7dfbaa255ace4c85844a78a67a38354c P 22a2a137f0f34c4590acc11d8139de50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5131ae7792e44f7f974f659ae02595da" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 43281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:17:34.380775 15755 mvcc.cc:205] Tried to move back new op lower bound from 7333492549120401408 to 7333492548983488512. Current Snapshot: MvccSnapshot[applied={T|T < 7333492549120401408}]
I20260926 06:17:34.388293 15240 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42998:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:17:34.388417 15240 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42998:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:17:34.389400 15240 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eb2e6df8b9ce4393bbb8e7f8dd777b37: Sending DeleteTablet for 3 replicas of tablet 7dfbaa255ace4c85844a78a67a38354c
I20260926 06:17:34.389840 15641 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dfbaa255ace4c85844a78a67a38354c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:17:34 UTC) from {username='kudu', principal='kudu/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:44544
I20260926 06:17:34.389938 15507 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dfbaa255ace4c85844a78a67a38354c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:17:34 UTC) from {username='kudu', principal='kudu/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:51096
I20260926 06:17:34.389966 15373 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dfbaa255ace4c85844a78a67a38354c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:17:34 UTC) from {username='kudu', principal='kudu/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:39896
I20260926 06:17:34.390766 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 15279
I20260926 06:17:34.397528 15764 tablet_replica.cc:330] T 7dfbaa255ace4c85844a78a67a38354c P 5131ae7792e44f7f974f659ae02595da: stopping tablet replica
I20260926 06:17:34.397696 15764 raft_consensus.cc:2229] T 7dfbaa255ace4c85844a78a67a38354c P 5131ae7792e44f7f974f659ae02595da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:17:34.397821 15764 raft_consensus.cc:2258] T 7dfbaa255ace4c85844a78a67a38354c P 5131ae7792e44f7f974f659ae02595da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:17:34.398355 15764 ts_tablet_manager.cc:1905] T 7dfbaa255ace4c85844a78a67a38354c P 5131ae7792e44f7f974f659ae02595da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:17:34.400955 15762 tablet_replica.cc:330] T 7dfbaa255ace4c85844a78a67a38354c P 22a2a137f0f34c4590acc11d8139de50: stopping tablet replica
I20260926 06:17:34.401111 15762 raft_consensus.cc:2229] T 7dfbaa255ace4c85844a78a67a38354c P 22a2a137f0f34c4590acc11d8139de50 [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:17:34.401264 15762 raft_consensus.cc:2258] T 7dfbaa255ace4c85844a78a67a38354c P 22a2a137f0f34c4590acc11d8139de50 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 06:17:34.401516 15227 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.110.193:40231 (error 108)
W20260926 06:17:34.401856 15225 connection.cc:535] server connection from 127.2.110.193:42101 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 06:17:34.405553 15764 ts_tablet_manager.cc:1918] T 7dfbaa255ace4c85844a78a67a38354c P 5131ae7792e44f7f974f659ae02595da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:17:34.405673 15764 log.cc:1196] T 7dfbaa255ace4c85844a78a67a38354c P 5131ae7792e44f7f974f659ae02595da: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-1/wal/wals/7dfbaa255ace4c85844a78a67a38354c
I20260926 06:17:34.406018 15764 ts_tablet_manager.cc:1939] T 7dfbaa255ace4c85844a78a67a38354c P 5131ae7792e44f7f974f659ae02595da: Deleting consensus metadata
I20260926 06:17:34.406297 15762 ts_tablet_manager.cc:1905] T 7dfbaa255ace4c85844a78a67a38354c P 22a2a137f0f34c4590acc11d8139de50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:17:34.406569 15227 catalog_manager.cc:4841] TS 5131ae7792e44f7f974f659ae02595da (127.2.110.194:43281): tablet 7dfbaa255ace4c85844a78a67a38354c (table test-table [id=7ca51787286d4129943c8ad30487256e]) successfully deleted
I20260926 06:17:34.408308 15762 ts_tablet_manager.cc:1918] T 7dfbaa255ace4c85844a78a67a38354c P 22a2a137f0f34c4590acc11d8139de50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:17:34.408444 15762 log.cc:1196] T 7dfbaa255ace4c85844a78a67a38354c P 22a2a137f0f34c4590acc11d8139de50: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403292369990-2491-0/minicluster-data/ts-2/wal/wals/7dfbaa255ace4c85844a78a67a38354c
I20260926 06:17:34.408867 15762 ts_tablet_manager.cc:1939] T 7dfbaa255ace4c85844a78a67a38354c P 22a2a137f0f34c4590acc11d8139de50: Deleting consensus metadata
I20260926 06:17:34.409390 15227 catalog_manager.cc:4841] TS 22a2a137f0f34c4590acc11d8139de50 (127.2.110.195:42875): tablet 7dfbaa255ace4c85844a78a67a38354c (table test-table [id=7ca51787286d4129943c8ad30487256e]) successfully deleted
I20260926 06:17:34.409508 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 15443
I20260926 06:17:34.416749 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 15576
I20260926 06:17:34.418712 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:34.418563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.193:40231 (local address 127.0.0.1:39906)
0926 06:17:34.418622 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:34.418670 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.193:40231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 06:17:34.419992 15227 catalog_manager.cc:4584] TS 61a97084ca5945f889a2aa4893685ed5 (127.2.110.193:40231): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7dfbaa255ace4c85844a78a67a38354c: Network error: Client connection negotiation failed: client connection to 127.2.110.193:40231: connect: Connection refused (error 111)
I20260926 06:17:34.424618 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 15208
I20260926 06:17:34.431270 2491 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 06:17:34.776665 2491 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 06:17:34.776777 2491 mini_ranger.cc:66] Stopping Ranger...
I20260926 06:17:35.122661 2491 mini_ranger.cc:68] Stopped Ranger
2026-09-26 06:17:35.122 UTC [14274] LOG: received smart shutdown request
2026-09-26 06:17:35.126 UTC [14274] LOG: background worker "logical replication launcher" (PID 14283) exited with exit code 1
2026-09-26 06:17:35.127 UTC [14278] LOG: shutting down
2026-09-26 06:17:35.306 UTC [14274] LOG: database system is shut down
2026-09-26T06:17:35Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53110 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:17:35 dist-test-slave-hndd krb5kdc[15775](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:17:35 dist-test-slave-hndd krb5kdc[15775](info): set up 2 sockets
Sep 26 06:17:35 dist-test-slave-hndd krb5kdc[15775](info): commencing operation
krb5kdc: starting...
W20260926 06:17:36.493263 2491 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Sep 26 06:17:36 dist-test-slave-hndd krb5kdc[15775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:17:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:17:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
I20260926 06:17:36.616927 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:32903
--webserver_interface=127.2.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:34835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.110.254:32903,127.2.110.253:35163,127.2.110.252:38677
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:17:36.705839 15788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:17:36.706063 15788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:17:36.706087 15788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:17:36.707195 15788 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:17:36.707231 15788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:17:36.707247 15788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:17:36.707259 15788 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:17:36.707273 15788 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:17:36.708768 15788 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:34835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.110.254:32903,127.2.110.253:35163,127.2.110.252:38677
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:32903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:17:36.708973 15788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:17:36.709160 15788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:17:36.714210 15793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:17:36.714217 15794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:17:36.714210 15796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:17:36.714684 15788 server_base.cc:1034] running on GCE node
I20260926 06:17:36.714951 15788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:17:36.715333 15788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:17:36.716542 15788 hybrid_clock.cc:648] HybridClock initialized: now 1790403456716525 us; error 31 us; skew 500 ppm
Sep 26 06:17:36 dist-test-slave-hndd krb5kdc[15775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403456, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:17:36.719201 15788 init.cc:377] Logged in from keytab as kudu/127.2.110.254@KRBTEST.COM (short username kudu)
I20260926 06:17:36.719692 15788 webserver.cc:460] Webserver started at http://127.2.110.254:41403/ using document root <none> and password file <none>
I20260926 06:17:36.719969 15788 fs_manager.cc:359] Metadata directory not provided
I20260926 06:17:36.720045 15788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:17:36.720204 15788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:17:36.721387 15788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/minicluster-data/master-0/data/instance:
uuid: "9517bfa26b9e485d88b9ac894a4b73b7"
format_stamp: "Formatted at 2026-09-26 06:17:36 on dist-test-slave-hndd"
server_key: "1a796e52bdf853abb5a1fb6c2b7c55ff"
server_key_iv: "53366cc8916801bd375c7e378b785a30"
server_key_version: "encryptionkey@0"
I20260926 06:17:36.721814 15788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/minicluster-data/master-0/wal/instance:
uuid: "9517bfa26b9e485d88b9ac894a4b73b7"
format_stamp: "Formatted at 2026-09-26 06:17:36 on dist-test-slave-hndd"
server_key: "1a796e52bdf853abb5a1fb6c2b7c55ff"
server_key_iv: "53366cc8916801bd375c7e378b785a30"
server_key_version: "encryptionkey@0"
I20260926 06:17:36.723224 15788 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 06:17:36.724164 15803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:17:36.724473 15788 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:17:36.724611 15788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "9517bfa26b9e485d88b9ac894a4b73b7"
format_stamp: "Formatted at 2026-09-26 06:17:36 on dist-test-slave-hndd"
server_key: "1a796e52bdf853abb5a1fb6c2b7c55ff"
server_key_iv: "53366cc8916801bd375c7e378b785a30"
server_key_version: "encryptionkey@0"
I20260926 06:17:36.724699 15788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:17:36.732256 15788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:17:36.734999 15788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:17:36.735194 15788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:17:36.739375 15788 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.254:32903
I20260926 06:17:36.739470 15855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.254:32903 every 8 connection(s)
I20260926 06:17:36.739837 15788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:17:36.740747 15856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:17:36.742218 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 15788
I20260926 06:17:36.742339 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/minicluster-data/master-0/wal/instance
I20260926 06:17:36.742558 2491 external_mini_cluster.cc:1428] Setting key 3053447897d279819f8bd14601567fd5
I20260926 06:17:36.742704 15856 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.110.254" port: 32903 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.110.253@KRBTEST.COM; defaulting to no policy
Sep 26 06:17:36 dist-test-slave-hndd krb5kdc[15775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403456, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.254@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.253@KRBTEST.COM" created.
I20260926 06:17:36.754864 15858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:36.743612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:32903 (local address 127.0.0.1:35982)
0926 06:17:36.743862 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:36.743911 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 06:17:36.744703 (+ 792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:17:36.745030 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:17:36.745033 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:17:36.745477 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:17:36.745653 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:36.745658 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:36.747294 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:36.747295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:36.748250 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:36.748254 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:36.749068 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:36.749070 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:36.749132 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:36.749755 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:17:36.749764 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:17:36.751414 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:17:36.753144 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:17:36.753150 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:17:36.753153 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:17:36.753455 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:17:36.753734 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:17:36.753735 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:17:36.753735 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:17:36.753842 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:17:36.754351 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:17:36.754357 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:17:36.754420 (+ 63us) client_negotiation.cc:769] Sending connection context
0926 06:17:36.754514 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 06:17:36.754650 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":100,"threads_started":1}
I20260926 06:17:36.754956 15859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:36.743682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35982 (local address 127.2.110.254:32903)
0926 06:17:36.743855 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 06:17:36.743866 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 06:17:36.744039 (+ 173us) server_negotiation.cc:407] Connection header received
0926 06:17:36.744873 (+ 834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:17:36.744875 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:17:36.744894 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:17:36.744984 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:17:36.745760 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:36.747152 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:36.748388 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:36.748941 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:36.749093 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:36.751511 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:17:36.751518 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:17:36.751519 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:17:36.751525 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:17:36.753065 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:36.753565 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:36.753569 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:36.753569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:36.753659 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:36.753925 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:36.753926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:36.753926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:36.754239 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:17:36.754337 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 06:17:36.754587 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 06:17:36.754743 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":130,"thread_start_us":80,"threads_started":1}
I20260926 06:17:36.755951 15856 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35163 } has no permanent_uuid. Determining permanent_uuid...
I20260926 06:17:36.756454 15858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:36.756300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.253:35163 (local address 127.0.0.1:41516)
0926 06:17:36.756358 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:36.756429 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.253:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 06:17:36.756588 15806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.110.253:35163: connect: Connection refused (error 111)
I20260926 06:17:36.757057 15858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:36.756957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.253:35163 (local address 127.0.0.1:41522)
0926 06:17:36.757004 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:36.757033 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.253:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 06:17:36.757157 15856 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.110.253:35163: Network error: Client connection negotiation failed: client connection to 127.2.110.253:35163: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.253.keytab.
Entry for principal kudu/127.2.110.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.253.keytab.
I20260926 06:17:36.770462 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.253:35163
--webserver_interface=127.2.110.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.253.keytab
--principal=kudu/127.2.110.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:34835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.110.254:32903,127.2.110.253:35163,127.2.110.252:38677
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 06:17:36.806470 15856 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35163 } attempt: 1
I20260926 06:17:36.807175 15858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:36.806985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.253:35163 (local address 127.0.0.1:41536)
0926 06:17:36.807060 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:36.807109 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.253:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 06:17:36.807554 15858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:36.807456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.253:35163 (local address 127.0.0.1:41546)
0926 06:17:36.807504 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:36.807537 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.253:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 06:17:36.807672 15856 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.110.253:35163: Network error: Client connection negotiation failed: client connection to 127.2.110.253:35163: connect: Connection refused (error 111)
W20260926 06:17:36.854499 15863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:17:36.854780 15863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:17:36.854823 15863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:17:36.856041 15863 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:17:36.856132 15863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:17:36.856154 15863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:17:36.856177 15863 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:17:36.856206 15863 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:17:36.857831 15863 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:34835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.110.254:32903,127.2.110.253:35163,127.2.110.252:38677
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.253.keytab
--principal=kudu/127.2.110.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.253:35163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.110.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:17:36.858141 15863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:17:36.858381 15863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:17:36.863636 15869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:17:36.863629 15871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:17:36.863878 15863 server_base.cc:1034] running on GCE node
W20260926 06:17:36.863629 15868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:17:36.864202 15863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:17:36.864563 15863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:17:36.865737 15863 hybrid_clock.cc:648] HybridClock initialized: now 1790403456865710 us; error 41 us; skew 500 ppm
Sep 26 06:17:36 dist-test-slave-hndd krb5kdc[15775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403456, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:17:36.868443 15863 init.cc:377] Logged in from keytab as kudu/127.2.110.253@KRBTEST.COM (short username kudu)
I20260926 06:17:36.868944 15863 webserver.cc:460] Webserver started at http://127.2.110.253:36671/ using document root <none> and password file <none>
I20260926 06:17:36.869195 15863 fs_manager.cc:359] Metadata directory not provided
I20260926 06:17:36.869267 15863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:17:36.869454 15863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:17:36.870558 15863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/minicluster-data/master-1/data/instance:
uuid: "11f2458eef524ea8b6ab014a70a9f238"
format_stamp: "Formatted at 2026-09-26 06:17:36 on dist-test-slave-hndd"
server_key: "f9a15a1288eb943132f4c94bb86f42a6"
server_key_iv: "6cc842fad38f38c00cf5454c3b46bb63"
server_key_version: "encryptionkey@0"
I20260926 06:17:36.870932 15863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/minicluster-data/master-1/wal/instance:
uuid: "11f2458eef524ea8b6ab014a70a9f238"
format_stamp: "Formatted at 2026-09-26 06:17:36 on dist-test-slave-hndd"
server_key: "f9a15a1288eb943132f4c94bb86f42a6"
server_key_iv: "6cc842fad38f38c00cf5454c3b46bb63"
server_key_version: "encryptionkey@0"
I20260926 06:17:36.872350 15863 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:17:36.873180 15878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:17:36.873427 15863 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:17:36.873545 15863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/minicluster-data/master-1/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/minicluster-data/master-1/wal
uuid: "11f2458eef524ea8b6ab014a70a9f238"
format_stamp: "Formatted at 2026-09-26 06:17:36 on dist-test-slave-hndd"
server_key: "f9a15a1288eb943132f4c94bb86f42a6"
server_key_iv: "6cc842fad38f38c00cf5454c3b46bb63"
server_key_version: "encryptionkey@0"
I20260926 06:17:36.873639 15863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:17:36.875835 15856 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35163 } attempt: 2
I20260926 06:17:36.876325 15858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:36.876178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.253:35163 (local address 127.0.0.1:41548)
0926 06:17:36.876248 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:36.876291 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.253:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 06:17:36.876721 15858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:36.876635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.253:35163 (local address 127.0.0.1:41560)
0926 06:17:36.876678 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:36.876704 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.253:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 06:17:36.876837 15856 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.110.253:35163: Network error: Client connection negotiation failed: client connection to 127.2.110.253:35163: connect: Connection refused (error 111)
I20260926 06:17:36.881884 15863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:17:36.884044 15863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:17:36.884186 15863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:17:36.888442 15863 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.253:35163
I20260926 06:17:36.888844 15863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/minicluster-data/master-1/data/info.pb
I20260926 06:17:36.888809 15930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.253:35163 every 8 connection(s)
I20260926 06:17:36.889639 15931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:17:36.891402 15931 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.110.254" port: 32903 } has no permanent_uuid. Determining permanent_uuid...
I20260926 06:17:36.896205 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 15863
I20260926 06:17:36.896332 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/minicluster-data/master-1/wal/instance
I20260926 06:17:36.896529 2491 external_mini_cluster.cc:1428] Setting key d38b7038a2c1be1b18dee3619245688c
Sep 26 06:17:36 dist-test-slave-hndd krb5kdc[15775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403456, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.253@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:17:36.902628 15859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:36.892229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35994 (local address 127.2.110.254:32903)
0926 06:17:36.892295 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 06:17:36.892297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:17:36.892463 (+ 166us) server_negotiation.cc:407] Connection header received
0926 06:17:36.892953 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:17:36.892955 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:17:36.892969 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:17:36.893036 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:17:36.893798 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:36.894291 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:36.895715 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:36.895920 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:36.895980 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:36.898797 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:17:36.898798 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:17:36.898799 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:17:36.898810 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:17:36.900648 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:36.901244 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:36.901246 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:36.901246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:36.901318 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:36.901822 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:36.901824 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:36.901824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:36.902011 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:17:36.902094 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 06:17:36.902384 (+ 290us) server_negotiation.cc:299] Negotiation successful
0926 06:17:36.902489 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 06:17:36.902670 15933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:36.892153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:32903 (local address 127.0.0.1:35994)
0926 06:17:36.892328 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:36.892370 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 06:17:36.892828 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:17:36.893114 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:17:36.893117 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:17:36.893467 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:17:36.893677 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:36.893681 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:36.894483 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:36.894485 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:36.895620 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:36.895625 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:36.896041 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:36.896043 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:36.896217 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:36.896858 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:17:36.896869 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:17:36.898682 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:17:36.900771 (+ 2089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:17:36.900784 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:17:36.900787 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:17:36.901150 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:17:36.901486 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:17:36.901488 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:17:36.901488 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:17:36.901730 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:17:36.902090 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:17:36.902095 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:17:36.902231 (+ 136us) client_negotiation.cc:769] Sending connection context
0926 06:17:36.902314 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 06:17:36.902480 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":102,"threads_started":1}
I20260926 06:17:36.903436 15931 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35163 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.110.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.252@KRBTEST.COM" created.
Sep 26 06:17:36 dist-test-slave-hndd krb5kdc[15775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403456, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.253@KRBTEST.COM for kudu/127.2.110.253@KRBTEST.COM
I20260926 06:17:36.911993 15933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:36.903736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.253:35163 (local address 127.0.0.1:41574)
0926 06:17:36.903793 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:36.903807 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 06:17:36.903848 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:17:36.904170 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:17:36.904171 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:17:36.904331 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:17:36.904404 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:36.904406 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:36.905287 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:36.905289 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:36.906016 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:36.906019 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:36.906527 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:36.906527 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:36.906583 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:36.907010 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:17:36.907015 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:17:36.909000 (+ 1985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:17:36.910889 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:17:36.910890 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:17:36.910891 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:17:36.911089 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:17:36.911356 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:17:36.911357 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:17:36.911357 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:17:36.911411 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:17:36.911747 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:17:36.911748 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:17:36.911788 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 06:17:36.911840 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 06:17:36.911920 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 06:17:36.912155 15935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:36.903818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41574 (local address 127.2.110.253:35163)
0926 06:17:36.904005 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 06:17:36.904010 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:17:36.904036 (+ 26us) server_negotiation.cc:407] Connection header received
0926 06:17:36.904055 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:17:36.904056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:17:36.904068 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:17:36.904143 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:17:36.904486 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:36.905192 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:36.906108 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:36.906467 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:36.906529 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:36.909094 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:17:36.909103 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:17:36.909103 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:17:36.909110 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:17:36.910798 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:36.911183 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:36.911186 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:36.911186 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:36.911252 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:36.911475 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:36.911476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:36.911477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:36.911680 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:17:36.911765 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 06:17:36.911901 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 06:17:36.912001 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":129,"threads_started":1}
I20260926 06:17:36.912766 15931 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.110.252" port: 38677 } has no permanent_uuid. Determining permanent_uuid...
I20260926 06:17:36.913169 15933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:36.913013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.252:38677 (local address 127.0.0.1:49578)
0926 06:17:36.913068 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:36.913124 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.252:38677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 06:17:36.913311 15881 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.110.252:38677: connect: Connection refused (error 111)
I20260926 06:17:36.913780 15933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:36.913684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.252:38677 (local address 127.0.0.1:49592)
0926 06:17:36.913735 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:36.913759 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.252:38677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 06:17:36.913892 15931 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.110.252:38677: Network error: Client connection negotiation failed: client connection to 127.2.110.252:38677: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.252.keytab.
Entry for principal kudu/127.2.110.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.252.keytab.
I20260926 06:17:36.924450 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.252:38677
--webserver_interface=127.2.110.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.252.keytab
--principal=kudu/127.2.110.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:34835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.110.254:32903,127.2.110.253:35163,127.2.110.252:38677
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 06:17:36.963137 15931 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.110.252" port: 38677 } attempt: 1
I20260926 06:17:36.963703 15933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:36.963527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.252:38677 (local address 127.0.0.1:49594)
0926 06:17:36.963599 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:36.963661 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.252:38677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 06:17:36.964197 15933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:36.964077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.252:38677 (local address 127.0.0.1:49600)
0926 06:17:36.964140 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:36.964165 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.252:38677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 06:17:36.964285 15931 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.110.252:38677: Network error: Client connection negotiation failed: client connection to 127.2.110.252:38677: connect: Connection refused (error 111)
I20260926 06:17:36.968024 15856 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35163 } attempt: 3
Sep 26 06:17:36 dist-test-slave-hndd krb5kdc[15775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403456, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.254@KRBTEST.COM for kudu/127.2.110.253@KRBTEST.COM
I20260926 06:17:36.975481 15858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:36.968419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.253:35163 (local address 127.0.0.1:41586)
0926 06:17:36.968482 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:36.968494 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:17:36.968547 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:17:36.968710 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:17:36.968711 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:17:36.968886 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:17:36.968953 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:36.968955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:36.969765 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:36.969766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:36.970287 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:36.970289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:36.970713 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:36.970714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:36.970797 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:36.971266 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:17:36.971274 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:17:36.972717 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:17:36.974423 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:17:36.974424 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:17:36.974425 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:17:36.974606 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:17:36.974811 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:17:36.974812 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:17:36.974813 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:17:36.974878 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:17:36.975145 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:17:36.975146 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:17:36.975195 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 06:17:36.975278 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 06:17:36.975384 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 06:17:36.975481 15935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:36.968505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41586 (local address 127.2.110.253:35163)
0926 06:17:36.968580 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 06:17:36.968582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:17:36.968593 (+ 11us) server_negotiation.cc:407] Connection header received
0926 06:17:36.968619 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:17:36.968620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:17:36.968630 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:17:36.968704 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:17:36.969063 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:36.969669 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:36.970396 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:36.970624 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:36.970713 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:36.972815 (+ 2102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:17:36.972816 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:17:36.972817 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:17:36.972824 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:17:36.974313 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:36.974681 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:36.974682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:36.974682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:36.974742 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:36.974950 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:36.974951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:36.974951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:36.975070 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:17:36.975143 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 06:17:36.975302 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 06:17:36.975381 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 06:17:36.976119 15856 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.110.252" port: 38677 } has no permanent_uuid. Determining permanent_uuid...
I20260926 06:17:36.976485 15858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:36.976380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.252:38677 (local address 127.0.0.1:49610)
0926 06:17:36.976431 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:36.976465 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.252:38677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 06:17:36.976817 15858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:36.976705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.252:38677 (local address 127.0.0.1:49624)
0926 06:17:36.976750 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:36.976773 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.252:38677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 06:17:36.976891 15856 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.110.252:38677: Network error: Client connection negotiation failed: client connection to 127.2.110.252:38677: connect: Connection refused (error 111)
I20260926 06:17:37.008044 15856 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.110.252" port: 38677 } attempt: 1
I20260926 06:17:37.008894 15858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:37.008426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.252:38677 (local address 127.0.0.1:49640)
0926 06:17:37.008644 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:37.008726 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.252:38677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176}
I20260926 06:17:37.009331 15858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:37.009212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.252:38677 (local address 127.0.0.1:49642)
0926 06:17:37.009267 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:37.009310 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.252:38677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 06:17:37.009483 15856 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.110.252:38677: Network error: Client connection negotiation failed: client connection to 127.2.110.252:38677: connect: Connection refused (error 111)
W20260926 06:17:37.009593 15938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:17:37.009824 15938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:17:37.009874 15938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:17:37.011121 15938 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:17:37.011195 15938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:17:37.011220 15938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:17:37.011247 15938 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:17:37.011278 15938 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:17:37.012887 15938 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:34835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.110.254:32903,127.2.110.253:35163,127.2.110.252:38677
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.252.keytab
--principal=kudu/127.2.110.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.252:38677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.110.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:17:37.013175 15938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:17:37.013438 15938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:17:37.018792 15946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:17:37.018796 15943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:17:37.019059 15938 server_base.cc:1034] running on GCE node
W20260926 06:17:37.018792 15944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:17:37.019366 15938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:17:37.019672 15938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:17:37.020855 15938 hybrid_clock.cc:648] HybridClock initialized: now 1790403457020833 us; error 44 us; skew 500 ppm
Sep 26 06:17:37 dist-test-slave-hndd krb5kdc[15775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403457, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:17:37.023921 15938 init.cc:377] Logged in from keytab as kudu/127.2.110.252@KRBTEST.COM (short username kudu)
I20260926 06:17:37.024492 15938 webserver.cc:460] Webserver started at http://127.2.110.252:33691/ using document root <none> and password file <none>
I20260926 06:17:37.024770 15938 fs_manager.cc:359] Metadata directory not provided
I20260926 06:17:37.024821 15938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:17:37.025027 15938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:17:37.026324 15938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/minicluster-data/master-2/data/instance:
uuid: "72220aa6f67f46c79c2893eefbbd046d"
format_stamp: "Formatted at 2026-09-26 06:17:37 on dist-test-slave-hndd"
server_key: "b84549152d080cec5f45e60e2a23c707"
server_key_iv: "9a82016d39b01d550c22f032ae014a56"
server_key_version: "encryptionkey@0"
I20260926 06:17:37.026741 15938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/minicluster-data/master-2/wal/instance:
uuid: "72220aa6f67f46c79c2893eefbbd046d"
format_stamp: "Formatted at 2026-09-26 06:17:37 on dist-test-slave-hndd"
server_key: "b84549152d080cec5f45e60e2a23c707"
server_key_iv: "9a82016d39b01d550c22f032ae014a56"
server_key_version: "encryptionkey@0"
I20260926 06:17:37.028584 15938 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260926 06:17:37.029450 15953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:17:37.029697 15938 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:17:37.029819 15938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/minicluster-data/master-2/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/minicluster-data/master-2/wal
uuid: "72220aa6f67f46c79c2893eefbbd046d"
format_stamp: "Formatted at 2026-09-26 06:17:37 on dist-test-slave-hndd"
server_key: "b84549152d080cec5f45e60e2a23c707"
server_key_iv: "9a82016d39b01d550c22f032ae014a56"
server_key_version: "encryptionkey@0"
I20260926 06:17:37.029896 15938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:17:37.032471 15931 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.110.252" port: 38677 } attempt: 2
I20260926 06:17:37.032946 15933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:37.032800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.252:38677 (local address 127.0.0.1:49648)
0926 06:17:37.032869 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:37.032913 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.252:38677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 06:17:37.033376 15933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:37.033238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.252:38677 (local address 127.0.0.1:49656)
0926 06:17:37.033299 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:37.033350 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.252:38677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 06:17:37.033489 15931 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.110.252:38677: Network error: Client connection negotiation failed: client connection to 127.2.110.252:38677: connect: Connection refused (error 111)
I20260926 06:17:37.046327 15938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:17:37.048738 15938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:17:37.048934 15938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:17:37.053175 15938 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.252:38677
I20260926 06:17:37.053210 16005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.252:38677 every 8 connection(s)
I20260926 06:17:37.053781 15938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/minicluster-data/master-2/data/info.pb
I20260926 06:17:37.054507 16006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:17:37.056141 16006 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.110.254" port: 32903 } has no permanent_uuid. Determining permanent_uuid...
I20260926 06:17:37.059756 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 15938
I20260926 06:17:37.059870 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403292369990-2491-0/minicluster-data/master-2/wal/instance
I20260926 06:17:37.060045 2491 external_mini_cluster.cc:1428] Setting key 926f633f072226c6756fcc240009ed2d
Sep 26 06:17:37 dist-test-slave-hndd krb5kdc[15775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403457, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.252@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
Sep 26 06:17:37 dist-test-slave-hndd krb5kdc[15775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.110.252@KRBTEST.COM
I20260926 06:17:37.067430 15859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:37.057021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36006 (local address 127.2.110.254:32903)
0926 06:17:37.057125 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 06:17:37.057127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:17:37.057424 (+ 297us) server_negotiation.cc:407] Connection header received
0926 06:17:37.057959 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:17:37.057960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:17:37.057971 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:17:37.058037 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:17:37.058745 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:37.059242 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:37.060729 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:37.061068 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:37.061130 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:37.063561 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:17:37.063563 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:17:37.063563 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:17:37.063569 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:17:37.065762 (+ 2193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:37.066267 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:37.066268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:37.066269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:37.066327 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:37.066674 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:37.066675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:37.066675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:37.066833 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:17:37.066911 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 06:17:37.067223 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 06:17:37.067307 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 06:17:37.067632 16008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:37.056945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:32903 (local address 127.0.0.1:36006)
0926 06:17:37.057206 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:37.057246 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 06:17:37.057837 (+ 591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:17:37.058117 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:17:37.058120 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:17:37.058405 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:17:37.058634 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:37.058638 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:37.059379 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:37.059380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:37.060628 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:37.060633 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:37.061159 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:37.061160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:37.061328 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:37.061976 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:17:37.061984 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:17:37.063463 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:17:37.065845 (+ 2382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:17:37.065855 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:17:37.065857 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:17:37.066193 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:17:37.066421 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:17:37.066422 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:17:37.066422 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:17:37.066588 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:17:37.066907 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:17:37.066913 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:17:37.067070 (+ 157us) client_negotiation.cc:769] Sending connection context
0926 06:17:37.067185 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 06:17:37.067339 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":132,"threads_started":1}
I20260926 06:17:37.068234 16006 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35163 } has no permanent_uuid. Determining permanent_uuid...
I20260926 06:17:37.069401 16010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:37.060483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49658 (local address 127.2.110.252:38677)
0926 06:17:37.060720 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 06:17:37.060725 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:17:37.060748 (+ 23us) server_negotiation.cc:407] Connection header received
0926 06:17:37.060772 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:17:37.060774 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:17:37.060795 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:17:37.060882 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:17:37.061641 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:37.062434 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:37.063249 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:37.063675 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:37.063728 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:37.066141 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:17:37.066147 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:17:37.066148 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:17:37.066155 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:17:37.068050 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:37.068398 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:37.068401 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:37.068402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:37.068465 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:37.068725 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:37.068726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:37.068726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:37.068923 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:17:37.069083 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 06:17:37.069167 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 06:17:37.069271 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":126,"threads_started":1}
Sep 26 06:17:37 dist-test-slave-hndd krb5kdc[15775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403457, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.252@KRBTEST.COM for kudu/127.2.110.253@KRBTEST.COM
Sep 26 06:17:37 dist-test-slave-hndd krb5kdc[15775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:17:37.076407 16008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:37.068506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.253:35163 (local address 127.0.0.1:41590)
0926 06:17:37.068725 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:37.068735 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:17:37.068768 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:17:37.068898 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:17:37.068898 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:17:37.069068 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:17:37.069137 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:37.069139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:37.070370 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:37.070371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:37.070873 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:37.070874 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:37.071274 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:37.071275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:37.071352 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:37.071790 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:17:37.071794 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:17:37.073048 (+ 1254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:17:37.075128 (+ 2080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:17:37.075129 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:17:37.075130 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:17:37.075360 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:17:37.075666 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:17:37.075667 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:17:37.075667 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:17:37.075723 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:17:37.076036 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:17:37.076037 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:17:37.076092 (+ 55us) client_negotiation.cc:769] Sending connection context
0926 06:17:37.076173 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 06:17:37.076275 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203}
I20260926 06:17:37.076434 15935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:37.068603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41590 (local address 127.2.110.253:35163)
0926 06:17:37.068649 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 06:17:37.068651 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:17:37.068779 (+ 128us) server_negotiation.cc:407] Connection header received
0926 06:17:37.068825 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:17:37.068826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:17:37.068838 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:17:37.068896 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:17:37.069237 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:37.069946 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:37.070966 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:37.071202 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:37.071256 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:37.073124 (+ 1868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:17:37.073125 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:17:37.073126 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:17:37.073132 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:17:37.075005 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:37.075510 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:37.075511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:37.075512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:37.075581 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:37.075805 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:37.075806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:37.075806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:37.075952 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:17:37.076007 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 06:17:37.076216 (+ 209us) server_negotiation.cc:299] Negotiation successful
0926 06:17:37.076330 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 06:17:37.076881 16006 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.110.252" port: 38677 } has no permanent_uuid. Determining permanent_uuid...
I20260926 06:17:37.078500 15859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:37.070339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36008 (local address 127.2.110.254:32903)
0926 06:17:37.070380 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 06:17:37.070381 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:17:37.070390 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:17:37.070450 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:17:37.070451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:17:37.070461 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:17:37.070505 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:17:37.071035 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:37.071614 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:37.072336 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:37.072548 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:37.072631 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:37.075079 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:17:37.075081 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:17:37.075081 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:17:37.075088 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:17:37.076948 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:37.077531 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:37.077532 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:37.077533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:37.077589 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:37.077897 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:37.077898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:37.077898 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:37.078055 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:17:37.078264 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0926 06:17:37.078318 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 06:17:37.078404 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Sep 26 06:17:37 dist-test-slave-hndd krb5kdc[15775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403457, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.252@KRBTEST.COM for kudu/127.2.110.252@KRBTEST.COM
Sep 26 06:17:37 dist-test-slave-hndd krb5kdc[15775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.110.253@KRBTEST.COM
I20260926 06:17:37.084830 16008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:37.077204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.252:38677 (local address 127.0.0.1:49664)
0926 06:17:37.077250 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:37.077260 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:17:37.077296 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:17:37.077489 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:17:37.077490 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:17:37.077595 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:17:37.077656 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:37.077658 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:37.078701 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:37.078702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:37.079320 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:37.079323 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:37.079666 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:37.079666 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:37.079712 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:37.080295 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:17:37.080300 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:17:37.081628 (+ 1328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:17:37.083391 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:17:37.083392 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:17:37.083393 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:17:37.083619 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:17:37.083846 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:17:37.083847 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:17:37.083847 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:17:37.083914 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:17:37.084243 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:17:37.084244 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:17:37.084286 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 06:17:37.084554 (+ 268us) client_negotiation.cc:240] Negotiation successful
0926 06:17:37.084679 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 06:17:37.084928 16010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:37.077305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49664 (local address 127.2.110.252:38677)
0926 06:17:37.077377 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 06:17:37.077379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:17:37.077387 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:17:37.077416 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:17:37.077417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:17:37.077434 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:17:37.077486 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:17:37.077892 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:37.078629 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:37.079416 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:37.079599 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:37.079670 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:37.081737 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:17:37.081738 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:17:37.081739 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:17:37.081746 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:17:37.083311 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:37.083724 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:37.083725 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:37.083725 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:37.083776 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:37.083988 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:37.083988 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:37.083989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:37.084139 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:17:37.084212 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 06:17:37.084657 (+ 445us) server_negotiation.cc:299] Negotiation successful
0926 06:17:37.084766 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 06:17:37.084657 15856 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.110.252" port: 38677 } attempt: 2
I20260926 06:17:37.088157 16006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72220aa6f67f46c79c2893eefbbd046d: Bootstrap starting.
Sep 26 06:17:37 dist-test-slave-hndd krb5kdc[15775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403456, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.254@KRBTEST.COM for kudu/127.2.110.252@KRBTEST.COM
I20260926 06:17:37.089648 15935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:37.079346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41592 (local address 127.2.110.253:35163)
0926 06:17:37.079387 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 06:17:37.079389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:17:37.079397 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:17:37.079439 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:17:37.079440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:17:37.079450 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:17:37.079505 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:17:37.079981 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:37.080431 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:37.081399 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:37.081625 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:37.081680 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:37.084517 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:17:37.084519 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:17:37.084519 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:17:37.084528 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:17:37.087813 (+ 3285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:37.088326 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:37.088328 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:37.088328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:37.088414 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:37.088677 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:37.088679 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:37.088679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:37.088866 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:17:37.089115 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0926 06:17:37.089419 (+ 304us) server_negotiation.cc:299] Negotiation successful
0926 06:17:37.089517 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 06:17:37.089650 16006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72220aa6f67f46c79c2893eefbbd046d: Neither blocks nor log segments found. Creating new log.
I20260926 06:17:37.090026 16006 log.cc:824] T 00000000000000000000000000000000 P 72220aa6f67f46c79c2893eefbbd046d: Log is configured to *not* fsync() on all Append() calls
I20260926 06:17:37.091086 16006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72220aa6f67f46c79c2893eefbbd046d: No bootstrap required, opened a new log
I20260926 06:17:37.091451 2491 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:17:37.093140 15858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:37.085269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.252:38677 (local address 127.0.0.1:49676)
0926 06:17:37.085472 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:37.085485 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:17:37.085643 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:17:37.086003 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:17:37.086004 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:17:37.086156 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:17:37.086198 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:37.086201 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:37.086964 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:37.086965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:37.087469 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:37.087470 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:37.087867 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:37.087868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:37.087929 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:37.088358 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:17:37.088363 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:17:37.089696 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:17:37.091784 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:17:37.091786 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:17:37.091787 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:17:37.092052 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:17:37.092305 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:17:37.092306 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:17:37.092307 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:17:37.092391 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:17:37.092773 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:17:37.092775 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:17:37.092833 (+ 58us) client_negotiation.cc:769] Sending connection context
0926 06:17:37.092910 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 06:17:37.093029 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185}
I20260926 06:17:37.093214 16010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:37.085386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49676 (local address 127.2.110.252:38677)
0926 06:17:37.085496 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 06:17:37.085498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:17:37.085647 (+ 149us) server_negotiation.cc:407] Connection header received
0926 06:17:37.085933 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:17:37.085933 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:17:37.085945 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:17:37.085997 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:17:37.086310 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:37.086854 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:37.087574 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:37.087780 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:37.087867 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:37.089818 (+ 1951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:17:37.089820 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:17:37.089820 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:17:37.089835 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:17:37.091683 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:37.092150 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:37.092151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:37.092152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:37.092221 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:37.092471 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:37.092472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:37.092473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:37.092667 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:17:37.092777 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 06:17:37.092952 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 06:17:37.093102 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 06:17:37.093689 16006 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72220aa6f67f46c79c2893eefbbd046d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9517bfa26b9e485d88b9ac894a4b73b7" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 32903 } } peers { permanent_uuid: "11f2458eef524ea8b6ab014a70a9f238" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35163 } } peers { permanent_uuid: "72220aa6f67f46c79c2893eefbbd046d" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 38677 } }
I20260926 06:17:37.094019 16006 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72220aa6f67f46c79c2893eefbbd046d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:17:37.094146 16006 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72220aa6f67f46c79c2893eefbbd046d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72220aa6f67f46c79c2893eefbbd046d, State: Initialized, Role: FOLLOWER
I20260926 06:17:37.094340 16006 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72220aa6f67f46c79c2893eefbbd046d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9517bfa26b9e485d88b9ac894a4b73b7" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 32903 } } peers { permanent_uuid: "11f2458eef524ea8b6ab014a70a9f238" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35163 } } peers { permanent_uuid: "72220aa6f67f46c79c2893eefbbd046d" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 38677 } }
I20260926 06:17:37.094823 16006 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72220aa6f67f46c79c2893eefbbd046d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:17:37.095121 15856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9517bfa26b9e485d88b9ac894a4b73b7: Bootstrap starting.
I20260926 06:17:37.095867 15856 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9517bfa26b9e485d88b9ac894a4b73b7: Neither blocks nor log segments found. Creating new log.
I20260926 06:17:37.096163 15856 log.cc:824] T 00000000000000000000000000000000 P 9517bfa26b9e485d88b9ac894a4b73b7: Log is configured to *not* fsync() on all Append() calls
I20260926 06:17:37.097033 15856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9517bfa26b9e485d88b9ac894a4b73b7: No bootstrap required, opened a new log
I20260926 06:17:37.098105 16019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72220aa6f67f46c79c2893eefbbd046d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9517bfa26b9e485d88b9ac894a4b73b7" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 32903 } } peers { permanent_uuid: "11f2458eef524ea8b6ab014a70a9f238" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35163 } } peers { permanent_uuid: "72220aa6f67f46c79c2893eefbbd046d" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 38677 } } }
I20260926 06:17:37.098228 16019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72220aa6f67f46c79c2893eefbbd046d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 06:17:37.099400 15856 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9517bfa26b9e485d88b9ac894a4b73b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9517bfa26b9e485d88b9ac894a4b73b7" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 32903 } } peers { permanent_uuid: "11f2458eef524ea8b6ab014a70a9f238" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35163 } } peers { permanent_uuid: "72220aa6f67f46c79c2893eefbbd046d" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 38677 } }
I20260926 06:17:37.099534 15856 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9517bfa26b9e485d88b9ac894a4b73b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:17:37.099579 15856 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9517bfa26b9e485d88b9ac894a4b73b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9517bfa26b9e485d88b9ac894a4b73b7, State: Initialized, Role: FOLLOWER
I20260926 06:17:37.099701 15856 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9517bfa26b9e485d88b9ac894a4b73b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9517bfa26b9e485d88b9ac894a4b73b7" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 32903 } } peers { permanent_uuid: "11f2458eef524ea8b6ab014a70a9f238" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35163 } } peers { permanent_uuid: "72220aa6f67f46c79c2893eefbbd046d" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 38677 } }
I20260926 06:17:37.100041 15856 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9517bfa26b9e485d88b9ac894a4b73b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:17:37.097057 15938 master_runner.cc:186] Error getting master registration for 127.2.110.254:32903: OK, instance_id { permanent_uuid: "9517bfa26b9e485d88b9ac894a4b73b7" instance_seqno: 1790403456738126 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 06:17:37.100762 15938 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 06:17:37.101429 16030 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 72220aa6f67f46c79c2893eefbbd046d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 06:17:37.101488 16030 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260926 06:17:37.102418 16044 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9517bfa26b9e485d88b9ac894a4b73b7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 06:17:37.102566 16044 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 06:17:37.102895 16032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9517bfa26b9e485d88b9ac894a4b73b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9517bfa26b9e485d88b9ac894a4b73b7" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 32903 } } peers { permanent_uuid: "11f2458eef524ea8b6ab014a70a9f238" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35163 } } peers { permanent_uuid: "72220aa6f67f46c79c2893eefbbd046d" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 38677 } } }
I20260926 06:17:37.103031 15859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:37.093024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36024 (local address 127.2.110.254:32903)
0926 06:17:37.093078 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 06:17:37.093080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:17:37.093416 (+ 336us) server_negotiation.cc:407] Connection header received
0926 06:17:37.093509 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:17:37.093510 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:17:37.093521 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:17:37.093553 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:17:37.094407 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:37.095177 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:37.096487 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:37.096786 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:37.096853 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:37.098982 (+ 2129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:17:37.098984 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:17:37.098984 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:17:37.098992 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:17:37.101423 (+ 2431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:37.101877 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:37.101878 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:37.101878 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:37.101942 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:37.102166 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:37.102167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:37.102167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:37.102317 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:17:37.102433 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 06:17:37.102845 (+ 412us) server_negotiation.cc:299] Negotiation successful
0926 06:17:37.102931 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 06:17:37.103029 16032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9517bfa26b9e485d88b9ac894a4b73b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 06:17:37.103530 15935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:37.093711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41606 (local address 127.2.110.253:35163)
0926 06:17:37.093756 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 06:17:37.093759 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:17:37.094473 (+ 714us) server_negotiation.cc:407] Connection header received
0926 06:17:37.094567 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:17:37.094568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:17:37.094581 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:17:37.094613 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:17:37.095344 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:37.096029 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:37.097397 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:37.097662 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:37.097704 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:37.099176 (+ 1472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:17:37.099178 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:17:37.099178 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:17:37.099185 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:17:37.100920 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:37.101431 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:37.101433 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:37.101433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:37.101493 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:37.101704 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:37.101705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:37.101706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:37.101857 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:17:37.101950 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 06:17:37.103335 (+ 1385us) server_negotiation.cc:299] Negotiation successful
0926 06:17:37.103426 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 06:17:37.103183 15788 master_runner.cc:186] Error getting master registration for 127.2.110.253:35163: OK, instance_id { permanent_uuid: "11f2458eef524ea8b6ab014a70a9f238" instance_seqno: 1790403456887160 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 06:17:37.107163 15788 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 06:17:37.109936 16010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:37.094044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49684 (local address 127.2.110.252:38677)
0926 06:17:37.094094 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 06:17:37.094096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:17:37.101766 (+ 7670us) server_negotiation.cc:407] Connection header received
0926 06:17:37.101896 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:17:37.101897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:17:37.101910 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:17:37.101945 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:17:37.102636 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:37.103558 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:37.104765 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:37.105082 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:37.105127 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:37.107110 (+ 1983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:17:37.107111 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:17:37.107112 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:17:37.107120 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:17:37.108681 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:37.109050 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:37.109051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:37.109051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:37.109105 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:37.109317 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:37.109318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:37.109319 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:37.109489 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:17:37.109597 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 06:17:37.109736 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 06:17:37.109825 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 06:17:37.124650 15931 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.110.252" port: 38677 } attempt: 3
Sep 26 06:17:37 dist-test-slave-hndd krb5kdc[15775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403456, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.253@KRBTEST.COM for kudu/127.2.110.252@KRBTEST.COM
I20260926 06:17:37.132220 16010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:37.125026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49696 (local address 127.2.110.252:38677)
0926 06:17:37.125084 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 06:17:37.125085 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:17:37.125094 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:17:37.125169 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:17:37.125170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:17:37.125187 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:17:37.125255 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:17:37.125610 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:37.126084 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:37.126820 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:37.127042 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:37.127128 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:37.129613 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:17:37.129615 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:17:37.129615 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:17:37.129623 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:17:37.131051 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:37.131404 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:37.131405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:37.131406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:37.131462 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:37.131667 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:37.131668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:37.131668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:37.131789 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:17:37.131858 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 06:17:37.132050 (+ 192us) server_negotiation.cc:299] Negotiation successful
0926 06:17:37.132123 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 06:17:37.132232 15933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:37.124981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.252:38677 (local address 127.0.0.1:49696)
0926 06:17:37.125048 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:37.125062 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 06:17:37.125108 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:17:37.125259 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:17:37.125259 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:17:37.125487 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:17:37.125534 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:37.125536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:37.126180 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:37.126181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:37.126744 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:37.126746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:37.127128 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:37.127128 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:37.127183 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:37.128182 (+ 999us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:17:37.128187 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:17:37.129523 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:17:37.131153 (+ 1630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:17:37.131154 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:17:37.131154 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:17:37.131330 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:17:37.131534 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:17:37.131535 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:17:37.131535 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:17:37.131595 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:17:37.131863 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:17:37.131864 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:17:37.131908 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 06:17:37.131987 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 06:17:37.132099 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 06:17:37.134217 15931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11f2458eef524ea8b6ab014a70a9f238: Bootstrap starting.
I20260926 06:17:37.134945 15931 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11f2458eef524ea8b6ab014a70a9f238: Neither blocks nor log segments found. Creating new log.
I20260926 06:17:37.135270 15931 log.cc:824] T 00000000000000000000000000000000 P 11f2458eef524ea8b6ab014a70a9f238: Log is configured to *not* fsync() on all Append() calls
I20260926 06:17:37.136075 15931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11f2458eef524ea8b6ab014a70a9f238: No bootstrap required, opened a new log
I20260926 06:17:37.138289 15931 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11f2458eef524ea8b6ab014a70a9f238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9517bfa26b9e485d88b9ac894a4b73b7" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 32903 } } peers { permanent_uuid: "11f2458eef524ea8b6ab014a70a9f238" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35163 } } peers { permanent_uuid: "72220aa6f67f46c79c2893eefbbd046d" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 38677 } }
I20260926 06:17:37.138430 15931 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11f2458eef524ea8b6ab014a70a9f238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:17:37.138494 15931 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11f2458eef524ea8b6ab014a70a9f238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11f2458eef524ea8b6ab014a70a9f238, State: Initialized, Role: FOLLOWER
I20260926 06:17:37.138653 15931 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11f2458eef524ea8b6ab014a70a9f238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9517bfa26b9e485d88b9ac894a4b73b7" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 32903 } } peers { permanent_uuid: "11f2458eef524ea8b6ab014a70a9f238" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35163 } } peers { permanent_uuid: "72220aa6f67f46c79c2893eefbbd046d" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 38677 } }
I20260926 06:17:37.138952 16047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11f2458eef524ea8b6ab014a70a9f238 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9517bfa26b9e485d88b9ac894a4b73b7" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 32903 } } peers { permanent_uuid: "11f2458eef524ea8b6ab014a70a9f238" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35163 } } peers { permanent_uuid: "72220aa6f67f46c79c2893eefbbd046d" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 38677 } } }
I20260926 06:17:37.139002 15931 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11f2458eef524ea8b6ab014a70a9f238 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:17:37.139096 16047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11f2458eef524ea8b6ab014a70a9f238 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 06:17:37.140923 16058 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 11f2458eef524ea8b6ab014a70a9f238: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 06:17:37.141039 16058 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 06:17:37.143817 15863 master_runner.cc:263] No leader master found from master 11f2458eef524ea8b6ab014a70a9f238
I20260926 06:17:37.143870 15863 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 06:17:37.156934 16047 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11f2458eef524ea8b6ab014a70a9f238 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:17:37.157074 16047 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11f2458eef524ea8b6ab014a70a9f238 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9517bfa26b9e485d88b9ac894a4b73b7" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 32903 } } peers { permanent_uuid: "11f2458eef524ea8b6ab014a70a9f238" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35163 } } peers { permanent_uuid: "72220aa6f67f46c79c2893eefbbd046d" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 38677 } }
I20260926 06:17:37.157393 16047 leader_election.cc:288] T 00000000000000000000000000000000 P 11f2458eef524ea8b6ab014a70a9f238 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9517bfa26b9e485d88b9ac894a4b73b7 (127.2.110.254:32903), 72220aa6f67f46c79c2893eefbbd046d (127.2.110.252:38677)
I20260926 06:17:37.157871 15980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "11f2458eef524ea8b6ab014a70a9f238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72220aa6f67f46c79c2893eefbbd046d" is_pre_election: true
I20260926 06:17:37.158038 15980 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 72220aa6f67f46c79c2893eefbbd046d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11f2458eef524ea8b6ab014a70a9f238 in term 0.
I20260926 06:17:37.158381 15881 leader_election.cc:302] T 00000000000000000000000000000000 P 11f2458eef524ea8b6ab014a70a9f238 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11f2458eef524ea8b6ab014a70a9f238, 72220aa6f67f46c79c2893eefbbd046d; no voters:
I20260926 06:17:37.158581 16047 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11f2458eef524ea8b6ab014a70a9f238 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:17:37.158633 16047 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11f2458eef524ea8b6ab014a70a9f238 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:17:37.158655 16047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11f2458eef524ea8b6ab014a70a9f238 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:17:37.159615 16047 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11f2458eef524ea8b6ab014a70a9f238 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9517bfa26b9e485d88b9ac894a4b73b7" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 32903 } } peers { permanent_uuid: "11f2458eef524ea8b6ab014a70a9f238" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35163 } } peers { permanent_uuid: "72220aa6f67f46c79c2893eefbbd046d" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 38677 } }
I20260926 06:17:37.159739 16047 leader_election.cc:288] T 00000000000000000000000000000000 P 11f2458eef524ea8b6ab014a70a9f238 [CANDIDATE]: Term 1 election: Requested vote from peers 9517bfa26b9e485d88b9ac894a4b73b7 (127.2.110.254:32903), 72220aa6f67f46c79c2893eefbbd046d (127.2.110.252:38677)
I20260926 06:17:37.160063 15980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "11f2458eef524ea8b6ab014a70a9f238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72220aa6f67f46c79c2893eefbbd046d"
I20260926 06:17:37.160154 15980 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72220aa6f67f46c79c2893eefbbd046d [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:17:37.161075 15980 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 72220aa6f67f46c79c2893eefbbd046d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11f2458eef524ea8b6ab014a70a9f238 in term 1.
I20260926 06:17:37.161314 15881 leader_election.cc:302] T 00000000000000000000000000000000 P 11f2458eef524ea8b6ab014a70a9f238 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11f2458eef524ea8b6ab014a70a9f238, 72220aa6f67f46c79c2893eefbbd046d; no voters:
I20260926 06:17:37.161755 15830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "11f2458eef524ea8b6ab014a70a9f238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9517bfa26b9e485d88b9ac894a4b73b7" is_pre_election: true
I20260926 06:17:37.161904 15830 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9517bfa26b9e485d88b9ac894a4b73b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11f2458eef524ea8b6ab014a70a9f238 in term 0.
I20260926 06:17:37.162026 15829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "11f2458eef524ea8b6ab014a70a9f238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9517bfa26b9e485d88b9ac894a4b73b7"
I20260926 06:17:37.162077 15829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9517bfa26b9e485d88b9ac894a4b73b7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:17:37.162951 15829 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9517bfa26b9e485d88b9ac894a4b73b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11f2458eef524ea8b6ab014a70a9f238 in term 1.
I20260926 06:17:37.163300 16047 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11f2458eef524ea8b6ab014a70a9f238 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:17:37.163468 16047 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11f2458eef524ea8b6ab014a70a9f238 [term 1 LEADER]: Becoming Leader. State: Replica: 11f2458eef524ea8b6ab014a70a9f238, State: Running, Role: LEADER
I20260926 06:17:37.163565 16047 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11f2458eef524ea8b6ab014a70a9f238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9517bfa26b9e485d88b9ac894a4b73b7" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 32903 } } peers { permanent_uuid: "11f2458eef524ea8b6ab014a70a9f238" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35163 } } peers { permanent_uuid: "72220aa6f67f46c79c2893eefbbd046d" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 38677 } }
I20260926 06:17:37.163959 16047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11f2458eef524ea8b6ab014a70a9f238 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11f2458eef524ea8b6ab014a70a9f238. Latest consensus state: current_term: 1 leader_uuid: "11f2458eef524ea8b6ab014a70a9f238" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9517bfa26b9e485d88b9ac894a4b73b7" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 32903 } } peers { permanent_uuid: "11f2458eef524ea8b6ab014a70a9f238" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35163 } } peers { permanent_uuid: "72220aa6f67f46c79c2893eefbbd046d" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 38677 } } }
I20260926 06:17:37.164256 16047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11f2458eef524ea8b6ab014a70a9f238 [sys.catalog]: This master's current role is: LEADER
I20260926 06:17:37.164986 16062 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:17:37.165206 16062 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:17:37.167897 15980 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 72220aa6f67f46c79c2893eefbbd046d [term 1 FOLLOWER]: Refusing update from remote peer 11f2458eef524ea8b6ab014a70a9f238: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:17:37.168308 16047 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 11f2458eef524ea8b6ab014a70a9f238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72220aa6f67f46c79c2893eefbbd046d" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 38677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:17:37.170020 16019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72220aa6f67f46c79c2893eefbbd046d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11f2458eef524ea8b6ab014a70a9f238. Latest consensus state: current_term: 1 leader_uuid: "11f2458eef524ea8b6ab014a70a9f238" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9517bfa26b9e485d88b9ac894a4b73b7" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 32903 } } peers { permanent_uuid: "11f2458eef524ea8b6ab014a70a9f238" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35163 } } peers { permanent_uuid: "72220aa6f67f46c79c2893eefbbd046d" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 38677 } } }
I20260926 06:17:37.170115 16019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72220aa6f67f46c79c2893eefbbd046d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 06:17:37.170643 15829 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9517bfa26b9e485d88b9ac894a4b73b7 [term 1 FOLLOWER]: Refusing update from remote peer 11f2458eef524ea8b6ab014a70a9f238: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:17:37.170925 16047 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 11f2458eef524ea8b6ab014a70a9f238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9517bfa26b9e485d88b9ac894a4b73b7" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 32903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:17:37.173074 16032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9517bfa26b9e485d88b9ac894a4b73b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11f2458eef524ea8b6ab014a70a9f238. Latest consensus state: current_term: 1 leader_uuid: "11f2458eef524ea8b6ab014a70a9f238" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9517bfa26b9e485d88b9ac894a4b73b7" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 32903 } } peers { permanent_uuid: "11f2458eef524ea8b6ab014a70a9f238" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35163 } } peers { permanent_uuid: "72220aa6f67f46c79c2893eefbbd046d" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 38677 } } }
I20260926 06:17:37.173166 16032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9517bfa26b9e485d88b9ac894a4b73b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 06:17:37.174633 16060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11f2458eef524ea8b6ab014a70a9f238 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "11f2458eef524ea8b6ab014a70a9f238" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9517bfa26b9e485d88b9ac894a4b73b7" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 32903 } } peers { permanent_uuid: "11f2458eef524ea8b6ab014a70a9f238" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35163 } } peers { permanent_uuid: "72220aa6f67f46c79c2893eefbbd046d" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 38677 } } }
I20260926 06:17:37.174741 16060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11f2458eef524ea8b6ab014a70a9f238 [sys.catalog]: This master's current role is: LEADER
I20260926 06:17:37.175542 16032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9517bfa26b9e485d88b9ac894a4b73b7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "11f2458eef524ea8b6ab014a70a9f238" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9517bfa26b9e485d88b9ac894a4b73b7" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 32903 } } peers { permanent_uuid: "11f2458eef524ea8b6ab014a70a9f238" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35163 } } peers { permanent_uuid: "72220aa6f67f46c79c2893eefbbd046d" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 38677 } } }
I20260926 06:17:37.175815 16032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9517bfa26b9e485d88b9ac894a4b73b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 06:17:37.176266 16063 mvcc.cc:205] Tried to move back new op lower bound from 7333492560553046016 to 7333492560542531584. Current Snapshot: MvccSnapshot[applied={T|T < 7333492560553046016}]
I20260926 06:17:37.176563 16066 mvcc.cc:205] Tried to move back new op lower bound from 7333492560553046016 to 7333492560542531584. Current Snapshot: MvccSnapshot[applied={T|T < 7333492560553046016}]
I20260926 06:17:37.177713 16047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11f2458eef524ea8b6ab014a70a9f238 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "11f2458eef524ea8b6ab014a70a9f238" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9517bfa26b9e485d88b9ac894a4b73b7" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 32903 } } peers { permanent_uuid: "11f2458eef524ea8b6ab014a70a9f238" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35163 } } peers { permanent_uuid: "72220aa6f67f46c79c2893eefbbd046d" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 38677 } } }
I20260926 06:17:37.177805 16047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11f2458eef524ea8b6ab014a70a9f238 [sys.catalog]: This master's current role is: LEADER
I20260926 06:17:37.178974 16019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72220aa6f67f46c79c2893eefbbd046d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "11f2458eef524ea8b6ab014a70a9f238" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9517bfa26b9e485d88b9ac894a4b73b7" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 32903 } } peers { permanent_uuid: "11f2458eef524ea8b6ab014a70a9f238" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 35163 } } peers { permanent_uuid: "72220aa6f67f46c79c2893eefbbd046d" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 38677 } } }
I20260926 06:17:37.179054 16019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72220aa6f67f46c79c2893eefbbd046d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 06:17:37.180152 16064 mvcc.cc:205] Tried to move back new op lower bound from 7333492560553046016 to 7333492560542531584. Current Snapshot: MvccSnapshot[applied={T|T < 7333492560553046016}]
I20260926 06:17:37.181485 16062 catalog_manager.cc:1347] Generated new cluster ID: 8057525c36cd446bbb56c8b9b3dbc533
I20260926 06:17:37.181541 16062 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:17:37.207531 16062 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:17:37.208357 16062 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:17:37.223973 16062 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11f2458eef524ea8b6ab014a70a9f238: Generated new TSK 0
I20260926 06:17:37.224315 16062 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:17:38.102008 16030 catalog_manager.cc:1259] Loaded cluster ID: 8057525c36cd446bbb56c8b9b3dbc533
I20260926 06:17:38.102094 16030 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 72220aa6f67f46c79c2893eefbbd046d: loading cluster ID for follower catalog manager: success
I20260926 06:17:38.103003 16044 catalog_manager.cc:1259] Loaded cluster ID: 8057525c36cd446bbb56c8b9b3dbc533
I20260926 06:17:38.103080 16044 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9517bfa26b9e485d88b9ac894a4b73b7: loading cluster ID for follower catalog manager: success
I20260926 06:17:38.103075 16030 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 72220aa6f67f46c79c2893eefbbd046d: acquiring CA information for follower catalog manager: success
I20260926 06:17:38.103384 16030 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 72220aa6f67f46c79c2893eefbbd046d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 06:17:38.103905 16044 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9517bfa26b9e485d88b9ac894a4b73b7: acquiring CA information for follower catalog manager: success
I20260926 06:17:38.104151 16044 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9517bfa26b9e485d88b9ac894a4b73b7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 06:17:38.292943 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 15788
I20260926 06:17:38.298986 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 15863
I20260926 06:17:38.305635 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 15938
2026-09-26T06:17:38Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2876 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790403292369990-2491-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:17:38 dist-test-slave-hndd krb5kdc[16078](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:17:38 dist-test-slave-hndd krb5kdc[16078](info): set up 2 sockets
Sep 26 06:17:38 dist-test-slave-hndd krb5kdc[16078](info): commencing operation
krb5kdc: starting...
W20260926 06:17:39.417750 2491 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Sep 26 06:17:39 dist-test-slave-hndd krb5kdc[16078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:17:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:17:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
I20260926 06:17:39.545655 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:43443
--webserver_interface=127.2.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:43405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.110.254:43443
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:17:39.630630 16091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:17:39.630898 16091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:17:39.630940 16091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:17:39.632107 16091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:17:39.632177 16091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:17:39.632205 16091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:17:39.632233 16091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:17:39.632264 16091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:17:39.633904 16091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:43405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790403292369990-2491-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.110.254:43443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:43443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:17:39.634182 16091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:17:39.634418 16091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:17:39.639966 16099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:17:39.639961 16096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:17:39.640003 16097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:17:39.640185 16091 server_base.cc:1034] running on GCE node
I20260926 06:17:39.640583 16091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:17:39.640893 16091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:17:39.642182 16091 hybrid_clock.cc:648] HybridClock initialized: now 1790403459642166 us; error 46 us; skew 500 ppm
Sep 26 06:17:39 dist-test-slave-hndd krb5kdc[16078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403459, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:17:39.649151 16091 init.cc:377] Logged in from keytab as kudu/127.2.110.254@KRBTEST.COM (short username kudu)
I20260926 06:17:39.649701 16091 webserver.cc:460] Webserver started at http://127.2.110.254:42775/ using document root <none> and password file <none>
I20260926 06:17:39.649946 16091 fs_manager.cc:359] Metadata directory not provided
I20260926 06:17:39.649996 16091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:17:39.650131 16091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:17:39.651270 16091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790403292369990-2491-0/minicluster-data/master-0/data/instance:
uuid: "cde9005eb37542a5b26f513f7736257e"
format_stamp: "Formatted at 2026-09-26 06:17:39 on dist-test-slave-hndd"
server_key: "b61b26e95da8e2f61c2feef2b6da1752"
server_key_iv: "0bb1e238b0756e007ce08f0e586f56ed"
server_key_version: "encryptionkey@0"
I20260926 06:17:39.651655 16091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790403292369990-2491-0/minicluster-data/master-0/wal/instance:
uuid: "cde9005eb37542a5b26f513f7736257e"
format_stamp: "Formatted at 2026-09-26 06:17:39 on dist-test-slave-hndd"
server_key: "b61b26e95da8e2f61c2feef2b6da1752"
server_key_iv: "0bb1e238b0756e007ce08f0e586f56ed"
server_key_version: "encryptionkey@0"
I20260926 06:17:39.653275 16091 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:17:39.654312 16106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:17:39.654816 16091 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:17:39.654956 16091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "cde9005eb37542a5b26f513f7736257e"
format_stamp: "Formatted at 2026-09-26 06:17:39 on dist-test-slave-hndd"
server_key: "b61b26e95da8e2f61c2feef2b6da1752"
server_key_iv: "0bb1e238b0756e007ce08f0e586f56ed"
server_key_version: "encryptionkey@0"
I20260926 06:17:39.655049 16091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:17:39.683091 16091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:17:39.685243 16091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:17:39.685431 16091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:17:39.689400 16091 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.254:43443
I20260926 06:17:39.689477 16158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.254:43443 every 8 connection(s)
I20260926 06:17:39.689827 16091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:17:39.690834 16159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:17:39.691396 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 16091
I20260926 06:17:39.691509 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790403292369990-2491-0/minicluster-data/master-0/wal/instance
I20260926 06:17:39.691694 2491 external_mini_cluster.cc:1428] Setting key 9c310cc37782c8dc3605c4d89cf03d78
I20260926 06:17:39.694134 16159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cde9005eb37542a5b26f513f7736257e: Bootstrap starting.
I20260926 06:17:39.694782 16159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cde9005eb37542a5b26f513f7736257e: Neither blocks nor log segments found. Creating new log.
I20260926 06:17:39.695104 16159 log.cc:824] T 00000000000000000000000000000000 P cde9005eb37542a5b26f513f7736257e: Log is configured to *not* fsync() on all Append() calls
I20260926 06:17:39.696031 16159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cde9005eb37542a5b26f513f7736257e: No bootstrap required, opened a new log
I20260926 06:17:39.698788 16159 raft_consensus.cc:348] T 00000000000000000000000000000000 P cde9005eb37542a5b26f513f7736257e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde9005eb37542a5b26f513f7736257e" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 43443 } }
I20260926 06:17:39.698939 16159 raft_consensus.cc:374] T 00000000000000000000000000000000 P cde9005eb37542a5b26f513f7736257e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:17:39.698962 16159 raft_consensus.cc:729] T 00000000000000000000000000000000 P cde9005eb37542a5b26f513f7736257e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cde9005eb37542a5b26f513f7736257e, State: Initialized, Role: FOLLOWER
Sep 26 06:17:39 dist-test-slave-hndd krb5kdc[16078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:17:39.699121 16159 consensus_queue.cc:260] T 00000000000000000000000000000000 P cde9005eb37542a5b26f513f7736257e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde9005eb37542a5b26f513f7736257e" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 43443 } }
I20260926 06:17:39.699226 16159 raft_consensus.cc:388] T 00000000000000000000000000000000 P cde9005eb37542a5b26f513f7736257e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:17:39.699272 16159 raft_consensus.cc:482] T 00000000000000000000000000000000 P cde9005eb37542a5b26f513f7736257e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:17:39.699324 16159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cde9005eb37542a5b26f513f7736257e [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:17:39.700148 16159 raft_consensus.cc:504] T 00000000000000000000000000000000 P cde9005eb37542a5b26f513f7736257e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde9005eb37542a5b26f513f7736257e" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 43443 } }
I20260926 06:17:39.700273 16159 leader_election.cc:302] T 00000000000000000000000000000000 P cde9005eb37542a5b26f513f7736257e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cde9005eb37542a5b26f513f7736257e; no voters:
I20260926 06:17:39.700485 16159 leader_election.cc:288] T 00000000000000000000000000000000 P cde9005eb37542a5b26f513f7736257e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:17:39.700599 16164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cde9005eb37542a5b26f513f7736257e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:17:39.700780 16159 sys_catalog.cc:564] T 00000000000000000000000000000000 P cde9005eb37542a5b26f513f7736257e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:17:39.700802 16164 raft_consensus.cc:686] T 00000000000000000000000000000000 P cde9005eb37542a5b26f513f7736257e [term 1 LEADER]: Becoming Leader. State: Replica: cde9005eb37542a5b26f513f7736257e, State: Running, Role: LEADER
I20260926 06:17:39.701021 16164 consensus_queue.cc:237] T 00000000000000000000000000000000 P cde9005eb37542a5b26f513f7736257e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde9005eb37542a5b26f513f7736257e" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 43443 } }
I20260926 06:17:39.702101 16166 sys_catalog.cc:455] T 00000000000000000000000000000000 P cde9005eb37542a5b26f513f7736257e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cde9005eb37542a5b26f513f7736257e. Latest consensus state: current_term: 1 leader_uuid: "cde9005eb37542a5b26f513f7736257e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde9005eb37542a5b26f513f7736257e" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 43443 } } }
I20260926 06:17:39.702189 16166 sys_catalog.cc:458] T 00000000000000000000000000000000 P cde9005eb37542a5b26f513f7736257e [sys.catalog]: This master's current role is: LEADER
I20260926 06:17:39.702253 16165 sys_catalog.cc:455] T 00000000000000000000000000000000 P cde9005eb37542a5b26f513f7736257e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cde9005eb37542a5b26f513f7736257e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde9005eb37542a5b26f513f7736257e" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 43443 } } }
I20260926 06:17:39.702344 16165 sys_catalog.cc:458] T 00000000000000000000000000000000 P cde9005eb37542a5b26f513f7736257e [sys.catalog]: This master's current role is: LEADER
W20260926 06:17:39.702913 16179 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cde9005eb37542a5b26f513f7736257e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 06:17:39.702976 16179 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 06:17:39.703130 16178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:17:39.703279 16178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:17:39.704030 16162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:39.692221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59430 (local address 127.2.110.254:43443)
0926 06:17:39.692428 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 06:17:39.692436 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 06:17:39.692462 (+ 26us) server_negotiation.cc:407] Connection header received
0926 06:17:39.692875 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:17:39.692882 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:17:39.692927 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:17:39.693045 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:17:39.693909 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:39.694972 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:39.696054 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:39.696530 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:39.696619 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:39.699604 (+ 2985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:17:39.699609 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:17:39.699610 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:17:39.699624 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:17:39.702062 (+ 2438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:39.702512 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:39.702515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:39.702515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:39.702617 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:39.702847 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:39.702848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:39.702849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:39.703142 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:17:39.703316 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 06:17:39.703681 (+ 365us) server_negotiation.cc:299] Negotiation successful
0926 06:17:39.703805 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":109,"threads_started":1}
I20260926 06:17:39.705001 16178 catalog_manager.cc:1347] Generated new cluster ID: 2eda3aebe0c340baa81fbe0361882a38
I20260926 06:17:39.705056 16178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:17:39.705395 2491 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:17:39.722452 16178 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:17:39.723038 16178 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:17:39.731659 16178 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cde9005eb37542a5b26f513f7736257e: Generated new TSK 0
I20260926 06:17:39.731882 16178 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:17:39.759914 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 16091
2026-09-26T06:17:39Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1451 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:17:39 dist-test-slave-hndd krb5kdc[16186](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:17:39 dist-test-slave-hndd krb5kdc[16186](info): set up 2 sockets
Sep 26 06:17:39 dist-test-slave-hndd krb5kdc[16186](info): commencing operation
krb5kdc: starting...
W20260926 06:17:40.813546 2491 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Sep 26 06:17:40 dist-test-slave-hndd krb5kdc[16186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:17:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:17:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
I20260926 06:17:40.944137 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:35291
--webserver_interface=127.2.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:43301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.110.254:35291
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:17:41.028015 16199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:17:41.028273 16199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:17:41.028322 16199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:17:41.029554 16199 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:17:41.029624 16199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:17:41.029649 16199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:17:41.029675 16199 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:17:41.029707 16199 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:17:41.031231 16199 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:43301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.110.254:35291
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:35291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:17:41.031556 16199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:17:41.031805 16199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:17:41.037072 16204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:17:41.037082 16207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:17:41.037112 16205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:17:41.037513 16199 server_base.cc:1034] running on GCE node
I20260926 06:17:41.037786 16199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:17:41.038090 16199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:17:41.039250 16199 hybrid_clock.cc:648] HybridClock initialized: now 1790403461039214 us; error 58 us; skew 500 ppm
Sep 26 06:17:41 dist-test-slave-hndd krb5kdc[16186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403461, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:17:41.041887 16199 init.cc:377] Logged in from keytab as kudu/127.2.110.254@KRBTEST.COM (short username kudu)
I20260926 06:17:41.042362 16199 webserver.cc:460] Webserver started at http://127.2.110.254:43605/ using document root <none> and password file <none>
I20260926 06:17:41.042613 16199 fs_manager.cc:359] Metadata directory not provided
I20260926 06:17:41.042680 16199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:17:41.042835 16199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:17:41.043912 16199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/master-0/data/instance:
uuid: "a58d836c12134ed295238497e9f8f94d"
format_stamp: "Formatted at 2026-09-26 06:17:41 on dist-test-slave-hndd"
server_key: "a7204bff9c35a98b89c459ba365ae40a"
server_key_iv: "6cc563169f4155333caa39a55ad03acf"
server_key_version: "encryptionkey@0"
I20260926 06:17:41.044294 16199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/master-0/wal/instance:
uuid: "a58d836c12134ed295238497e9f8f94d"
format_stamp: "Formatted at 2026-09-26 06:17:41 on dist-test-slave-hndd"
server_key: "a7204bff9c35a98b89c459ba365ae40a"
server_key_iv: "6cc563169f4155333caa39a55ad03acf"
server_key_version: "encryptionkey@0"
I20260926 06:17:41.045677 16199 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 06:17:41.046558 16214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:17:41.046756 16199 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:17:41.046875 16199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "a58d836c12134ed295238497e9f8f94d"
format_stamp: "Formatted at 2026-09-26 06:17:41 on dist-test-slave-hndd"
server_key: "a7204bff9c35a98b89c459ba365ae40a"
server_key_iv: "6cc563169f4155333caa39a55ad03acf"
server_key_version: "encryptionkey@0"
I20260926 06:17:41.046974 16199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:17:41.061005 16199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:17:41.063136 16199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:17:41.063324 16199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:17:41.067718 16199 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.254:35291
I20260926 06:17:41.067751 16266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.254:35291 every 8 connection(s)
I20260926 06:17:41.068133 16199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:17:41.069236 16267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:17:41.069276 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 16199
I20260926 06:17:41.069381 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/master-0/wal/instance
I20260926 06:17:41.069605 2491 external_mini_cluster.cc:1428] Setting key 8d0a61d5b61f83a1a3ee73901c70ce20
I20260926 06:17:41.072619 16267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d: Bootstrap starting.
I20260926 06:17:41.073407 16267 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d: Neither blocks nor log segments found. Creating new log.
I20260926 06:17:41.073706 16267 log.cc:824] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d: Log is configured to *not* fsync() on all Append() calls
I20260926 06:17:41.074405 16267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d: No bootstrap required, opened a new log
Sep 26 06:17:41 dist-test-slave-hndd krb5kdc[16186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:17:41.076997 16267 raft_consensus.cc:348] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58d836c12134ed295238497e9f8f94d" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 35291 } }
I20260926 06:17:41.077160 16267 raft_consensus.cc:374] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:17:41.077184 16267 raft_consensus.cc:729] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a58d836c12134ed295238497e9f8f94d, State: Initialized, Role: FOLLOWER
I20260926 06:17:41.077325 16267 consensus_queue.cc:260] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58d836c12134ed295238497e9f8f94d" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 35291 } }
I20260926 06:17:41.077497 16267 raft_consensus.cc:388] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:17:41.077545 16267 raft_consensus.cc:482] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:17:41.077598 16267 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:17:41.078384 16267 raft_consensus.cc:504] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58d836c12134ed295238497e9f8f94d" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 35291 } }
I20260926 06:17:41.078504 16267 leader_election.cc:302] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a58d836c12134ed295238497e9f8f94d; no voters:
I20260926 06:17:41.078723 16267 leader_election.cc:288] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:17:41.078759 16272 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:17:41.078862 16272 raft_consensus.cc:686] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d [term 1 LEADER]: Becoming Leader. State: Replica: a58d836c12134ed295238497e9f8f94d, State: Running, Role: LEADER
I20260926 06:17:41.078998 16267 sys_catalog.cc:564] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:17:41.078975 16272 consensus_queue.cc:237] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58d836c12134ed295238497e9f8f94d" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 35291 } }
I20260926 06:17:41.079357 16272 sys_catalog.cc:455] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a58d836c12134ed295238497e9f8f94d. Latest consensus state: current_term: 1 leader_uuid: "a58d836c12134ed295238497e9f8f94d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58d836c12134ed295238497e9f8f94d" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 35291 } } }
I20260926 06:17:41.079463 16272 sys_catalog.cc:458] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d [sys.catalog]: This master's current role is: LEADER
I20260926 06:17:41.079373 16273 sys_catalog.cc:455] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a58d836c12134ed295238497e9f8f94d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58d836c12134ed295238497e9f8f94d" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 35291 } } }
I20260926 06:17:41.079739 16278 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:17:41.080478 16278 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:17:41.081189 16273 sys_catalog.cc:458] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d [sys.catalog]: This master's current role is: LEADER
I20260926 06:17:41.081415 16270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:41.070196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44276 (local address 127.2.110.254:35291)
0926 06:17:41.070380 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 06:17:41.070396 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 06:17:41.070428 (+ 32us) server_negotiation.cc:407] Connection header received
0926 06:17:41.070905 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:17:41.070911 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:17:41.070959 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:17:41.071081 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:17:41.071897 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:41.072860 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:41.074015 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:41.074426 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:41.074563 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:41.077309 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:17:41.077313 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:17:41.077314 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:17:41.077354 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:17:41.079556 (+ 2202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:41.079980 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:41.079983 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:41.079983 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:41.080085 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:41.080316 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:41.080317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:41.080317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:41.080721 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:17:41.080848 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 06:17:41.081002 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 06:17:41.081134 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":94,"threads_started":1}
I20260926 06:17:41.086762 16278 catalog_manager.cc:1347] Generated new cluster ID: c35f709572054fc4a7c26e4aa20b5718
I20260926 06:17:41.086844 16278 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:17:41.099483 16278 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:17:41.100234 16278 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:17:41.105156 16278 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d: Generated new TSK 0
I20260926 06:17:41.105386 16278 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab.
Entry for principal kudu/127.2.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab.
I20260926 06:17:41.141258 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.193:0
--local_ip_for_outbound_sockets=127.2.110.193
--webserver_interface=127.2.110.193
--webserver_port=0
--tserver_master_addrs=127.2.110.254:35291
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab
--principal=kudu/127.2.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:43301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:17:41.227391 16293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:17:41.227597 16293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:17:41.227623 16293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:17:41.228856 16293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:17:41.228905 16293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:17:41.228933 16293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.193
I20260926 06:17:41.230458 16293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:43301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab
--principal=kudu/127.2.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.110.193
--webserver_port=0
--tserver_master_addrs=127.2.110.254:35291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.193
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:17:41.230654 16293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:17:41.230836 16293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:17:41.236102 16299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:17:41.236096 16298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:17:41.236125 16301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:17:41.236120 16293 server_base.cc:1034] running on GCE node
I20260926 06:17:41.236537 16293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:17:41.236786 16293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:17:41.237952 16293 hybrid_clock.cc:648] HybridClock initialized: now 1790403461237944 us; error 50 us; skew 500 ppm
Sep 26 06:17:41 dist-test-slave-hndd krb5kdc[16186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403461, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:17:41.240521 16293 init.cc:377] Logged in from keytab as kudu/127.2.110.193@KRBTEST.COM (short username kudu)
I20260926 06:17:41.240965 16293 webserver.cc:460] Webserver started at http://127.2.110.193:44895/ using document root <none> and password file <none>
I20260926 06:17:41.241222 16293 fs_manager.cc:359] Metadata directory not provided
I20260926 06:17:41.241293 16293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:17:41.241475 16293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:17:41.242517 16293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-0/data/instance:
uuid: "8778addf8daf4894b885ea833fac225c"
format_stamp: "Formatted at 2026-09-26 06:17:41 on dist-test-slave-hndd"
server_key: "dbe7afcd73ffde686eb3744261deb453"
server_key_iv: "f2ae0f5d2705ce8bfce82f650158c135"
server_key_version: "encryptionkey@0"
I20260926 06:17:41.242880 16293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-0/wal/instance:
uuid: "8778addf8daf4894b885ea833fac225c"
format_stamp: "Formatted at 2026-09-26 06:17:41 on dist-test-slave-hndd"
server_key: "dbe7afcd73ffde686eb3744261deb453"
server_key_iv: "f2ae0f5d2705ce8bfce82f650158c135"
server_key_version: "encryptionkey@0"
I20260926 06:17:41.244221 16293 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:17:41.245127 16308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:17:41.245417 16293 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:17:41.245545 16293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-0/wal
uuid: "8778addf8daf4894b885ea833fac225c"
format_stamp: "Formatted at 2026-09-26 06:17:41 on dist-test-slave-hndd"
server_key: "dbe7afcd73ffde686eb3744261deb453"
server_key_iv: "f2ae0f5d2705ce8bfce82f650158c135"
server_key_version: "encryptionkey@0"
I20260926 06:17:41.245623 16293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:17:41.261454 16293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:17:41.263574 16293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:17:41.263731 16293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:17:41.264017 16293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:17:41.264351 16293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:17:41.264385 16293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:17:41.264443 16293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:17:41.264470 16293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:17:41.270409 16293 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.193:43911
I20260926 06:17:41.270481 16421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.193:43911 every 8 connection(s)
I20260926 06:17:41.270821 16293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
I20260926 06:17:41.276683 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 16293
I20260926 06:17:41.276810 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-0/wal/instance
I20260926 06:17:41.277031 2491 external_mini_cluster.cc:1428] Setting key f1cd85e759d5f44244995e684bf49e79
Sep 26 06:17:41 dist-test-slave-hndd krb5kdc[16186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403461, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.193@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:17:41.281545 16270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:41.271626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:56093 (local address 127.2.110.254:35291)
0926 06:17:41.271687 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 06:17:41.271689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:17:41.271896 (+ 207us) server_negotiation.cc:407] Connection header received
0926 06:17:41.272454 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:17:41.272456 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:17:41.272467 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:17:41.272530 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:17:41.273598 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:41.274120 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:41.275124 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:41.275337 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:41.275430 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:41.278185 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:17:41.278187 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:17:41.278187 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:17:41.278195 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:17:41.279805 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:41.280391 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:41.280392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:41.280392 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:41.280448 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:41.280771 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:41.280772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:41.280773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:41.280932 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:17:41.281009 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 06:17:41.281270 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 06:17:41.281392 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 06:17:41.281617 16424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:41.271612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.193:56093)
0926 06:17:41.271774 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:41.271810 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 06:17:41.272334 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:17:41.272601 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:17:41.272604 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:17:41.273209 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:17:41.273488 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:41.273494 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:41.274222 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:41.274223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:41.275028 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:41.275030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:41.275437 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:41.275439 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:41.275578 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:41.276139 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:17:41.276151 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:17:41.278092 (+ 1941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:17:41.279892 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:17:41.279900 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:17:41.279903 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:17:41.280299 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:17:41.280523 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:17:41.280524 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:17:41.280524 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:17:41.280685 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:17:41.281009 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:17:41.281014 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:17:41.281138 (+ 124us) client_negotiation.cc:769] Sending connection context
0926 06:17:41.281223 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 06:17:41.281426 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":73,"threads_started":1}
I20260926 06:17:41.282239 16422 heartbeater.cc:347] Connected to a master server at 127.2.110.254:35291
I20260926 06:17:41.282358 16422 heartbeater.cc:464] Registering TS with master...
I20260926 06:17:41.282558 16422 heartbeater.cc:511] Master 127.2.110.254:35291 requested a full tablet report, sending...
I20260926 06:17:41.283179 16231 ts_manager.cc:194] Registered new tserver with Master: 8778addf8daf4894b885ea833fac225c (127.2.110.193:43911)
I20260926 06:17:41.284196 16231 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.193@KRBTEST.COM'} at 127.2.110.193:56093
WARNING: no policy specified for kudu/127.2.110.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab.
Entry for principal kudu/127.2.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab.
I20260926 06:17:41.305042 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.194:0
--local_ip_for_outbound_sockets=127.2.110.194
--webserver_interface=127.2.110.194
--webserver_port=0
--tserver_master_addrs=127.2.110.254:35291
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab
--principal=kudu/127.2.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:43301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:17:41.388489 16427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:17:41.388718 16427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:17:41.388743 16427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:17:41.390030 16427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:17:41.390076 16427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:17:41.390105 16427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.194
I20260926 06:17:41.391846 16427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:43301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab
--principal=kudu/127.2.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.110.194
--webserver_port=0
--tserver_master_addrs=127.2.110.254:35291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.194
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:17:41.392092 16427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:17:41.392395 16427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:17:41.398046 16435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:17:41.398105 16432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:17:41.398049 16433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:17:41.398267 16427 server_base.cc:1034] running on GCE node
I20260926 06:17:41.398577 16427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:17:41.398841 16427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:17:41.400138 16427 hybrid_clock.cc:648] HybridClock initialized: now 1790403461400104 us; error 34 us; skew 500 ppm
Sep 26 06:17:41 dist-test-slave-hndd krb5kdc[16186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403461, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:17:41.403888 16427 init.cc:377] Logged in from keytab as kudu/127.2.110.194@KRBTEST.COM (short username kudu)
I20260926 06:17:41.404549 16427 webserver.cc:460] Webserver started at http://127.2.110.194:34939/ using document root <none> and password file <none>
I20260926 06:17:41.404793 16427 fs_manager.cc:359] Metadata directory not provided
I20260926 06:17:41.404855 16427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:17:41.404960 16427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:17:41.406257 16427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-1/data/instance:
uuid: "deaf547998bd4cce8d3970dce1eb51ad"
format_stamp: "Formatted at 2026-09-26 06:17:41 on dist-test-slave-hndd"
server_key: "c565334f1559a7d546f2685326074869"
server_key_iv: "e029a9d7333aa4497ca7145f386b377f"
server_key_version: "encryptionkey@0"
I20260926 06:17:41.406594 16427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-1/wal/instance:
uuid: "deaf547998bd4cce8d3970dce1eb51ad"
format_stamp: "Formatted at 2026-09-26 06:17:41 on dist-test-slave-hndd"
server_key: "c565334f1559a7d546f2685326074869"
server_key_iv: "e029a9d7333aa4497ca7145f386b377f"
server_key_version: "encryptionkey@0"
I20260926 06:17:41.408089 16427 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 06:17:41.408883 16442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:17:41.409102 16427 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:17:41.409196 16427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-1/wal
uuid: "deaf547998bd4cce8d3970dce1eb51ad"
format_stamp: "Formatted at 2026-09-26 06:17:41 on dist-test-slave-hndd"
server_key: "c565334f1559a7d546f2685326074869"
server_key_iv: "e029a9d7333aa4497ca7145f386b377f"
server_key_version: "encryptionkey@0"
I20260926 06:17:41.409260 16427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:17:41.424220 16427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:17:41.426708 16427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:17:41.426833 16427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:17:41.427039 16427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:17:41.427372 16427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:17:41.427412 16427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:17:41.427435 16427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:17:41.427448 16427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:17:41.434955 16427 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.194:46487
I20260926 06:17:41.435041 16555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.194:46487 every 8 connection(s)
I20260926 06:17:41.435478 16427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
I20260926 06:17:41.440353 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 16427
I20260926 06:17:41.440485 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-1/wal/instance
I20260926 06:17:41.440689 2491 external_mini_cluster.cc:1428] Setting key ef4f19653f738dff6cd842790c2d6243
Sep 26 06:17:41 dist-test-slave-hndd krb5kdc[16186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403461, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.194@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:17:41.447566 16270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:41.436470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:37191 (local address 127.2.110.254:35291)
0926 06:17:41.436584 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 06:17:41.436589 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:17:41.436851 (+ 262us) server_negotiation.cc:407] Connection header received
0926 06:17:41.437782 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:17:41.437783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:17:41.437799 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:17:41.437854 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:17:41.438874 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:41.439399 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:41.440644 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:41.440922 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:41.441019 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:41.444252 (+ 3233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:17:41.444254 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:17:41.444254 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:17:41.444263 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:17:41.445868 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:41.446411 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:41.446412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:41.446413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:41.446475 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:41.446814 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:41.446815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:41.446815 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:41.446948 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:17:41.447014 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 06:17:41.447420 (+ 406us) server_negotiation.cc:299] Negotiation successful
0926 06:17:41.447493 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 06:17:41.447883 16558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:41.436313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.194:37191)
0926 06:17:41.436639 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:41.436720 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0926 06:17:41.437528 (+ 808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:17:41.437944 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:17:41.437948 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:17:41.438548 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:17:41.438754 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:41.438760 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:41.439503 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:41.439504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:41.440513 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:41.440516 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:41.441146 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:41.441148 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:41.441421 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:41.442193 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:17:41.442207 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:17:41.444133 (+ 1926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:17:41.445959 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:17:41.445968 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:17:41.445971 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:17:41.446311 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:17:41.446560 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:17:41.446561 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:17:41.446561 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:17:41.446722 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:17:41.447028 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:17:41.447035 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:17:41.447233 (+ 198us) client_negotiation.cc:769] Sending connection context
0926 06:17:41.447372 (+ 139us) client_negotiation.cc:240] Negotiation successful
0926 06:17:41.447587 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":114,"threads_started":1}
I20260926 06:17:41.448676 16556 heartbeater.cc:347] Connected to a master server at 127.2.110.254:35291
I20260926 06:17:41.448786 16556 heartbeater.cc:464] Registering TS with master...
I20260926 06:17:41.449028 16556 heartbeater.cc:511] Master 127.2.110.254:35291 requested a full tablet report, sending...
I20260926 06:17:41.449568 16231 ts_manager.cc:194] Registered new tserver with Master: deaf547998bd4cce8d3970dce1eb51ad (127.2.110.194:46487)
I20260926 06:17:41.449986 16231 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.194@KRBTEST.COM'} at 127.2.110.194:37191
WARNING: no policy specified for kudu/127.2.110.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab.
Entry for principal kudu/127.2.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab.
I20260926 06:17:41.472363 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.195:0
--local_ip_for_outbound_sockets=127.2.110.195
--webserver_interface=127.2.110.195
--webserver_port=0
--tserver_master_addrs=127.2.110.254:35291
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab
--principal=kudu/127.2.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:43301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:17:41.561098 16561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:17:41.561473 16561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:17:41.561512 16561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:17:41.563277 16561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:17:41.563329 16561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:17:41.563364 16561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.195
I20260926 06:17:41.565065 16561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:43301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab
--principal=kudu/127.2.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.110.195
--webserver_port=0
--tserver_master_addrs=127.2.110.254:35291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.195
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:17:41.565305 16561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:17:41.565590 16561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:17:41.570873 16567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:17:41.570878 16569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:17:41.570971 16561 server_base.cc:1034] running on GCE node
W20260926 06:17:41.570919 16566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:17:41.571355 16561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:17:41.571640 16561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:17:41.572896 16561 hybrid_clock.cc:648] HybridClock initialized: now 1790403461572891 us; error 55 us; skew 500 ppm
Sep 26 06:17:41 dist-test-slave-hndd krb5kdc[16186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403461, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:17:41.575778 16561 init.cc:377] Logged in from keytab as kudu/127.2.110.195@KRBTEST.COM (short username kudu)
I20260926 06:17:41.576331 16561 webserver.cc:460] Webserver started at http://127.2.110.195:38621/ using document root <none> and password file <none>
I20260926 06:17:41.576696 16561 fs_manager.cc:359] Metadata directory not provided
I20260926 06:17:41.576753 16561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:17:41.577028 16561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:17:41.578755 16561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-2/data/instance:
uuid: "1c6be47ad34f432987a5d2ef28f0123f"
format_stamp: "Formatted at 2026-09-26 06:17:41 on dist-test-slave-hndd"
server_key: "62639a18bf85d59a91212d35060eb7a6"
server_key_iv: "08e8d15d0ea5c3388dbc3b4800c2ef5d"
server_key_version: "encryptionkey@0"
I20260926 06:17:41.579257 16561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-2/wal/instance:
uuid: "1c6be47ad34f432987a5d2ef28f0123f"
format_stamp: "Formatted at 2026-09-26 06:17:41 on dist-test-slave-hndd"
server_key: "62639a18bf85d59a91212d35060eb7a6"
server_key_iv: "08e8d15d0ea5c3388dbc3b4800c2ef5d"
server_key_version: "encryptionkey@0"
I20260926 06:17:41.581128 16561 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260926 06:17:41.582430 16576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:17:41.582773 16561 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:17:41.582909 16561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-2/wal
uuid: "1c6be47ad34f432987a5d2ef28f0123f"
format_stamp: "Formatted at 2026-09-26 06:17:41 on dist-test-slave-hndd"
server_key: "62639a18bf85d59a91212d35060eb7a6"
server_key_iv: "08e8d15d0ea5c3388dbc3b4800c2ef5d"
server_key_version: "encryptionkey@0"
I20260926 06:17:41.583001 16561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:17:41.601497 16561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:17:41.603868 16561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:17:41.604089 16561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:17:41.604354 16561 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:17:41.604763 16561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:17:41.604840 16561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:17:41.604876 16561 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:17:41.604907 16561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:17:41.611136 16561 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.195:33005
I20260926 06:17:41.611179 16689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.195:33005 every 8 connection(s)
I20260926 06:17:41.611522 16561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
I20260926 06:17:41.617877 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 16561
I20260926 06:17:41.617949 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-2/wal/instance
Sep 26 06:17:41 dist-test-slave-hndd krb5kdc[16186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403461, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.195@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:17:41.618131 2491 external_mini_cluster.cc:1428] Setting key 4849b03295afffb0bb0b071f2c249d8c
I20260926 06:17:41.621212 16270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:41.612315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:56319 (local address 127.2.110.254:35291)
0926 06:17:41.612406 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 06:17:41.612409 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:17:41.612575 (+ 166us) server_negotiation.cc:407] Connection header received
0926 06:17:41.613148 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:17:41.613149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:17:41.613159 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:17:41.613216 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:17:41.614318 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:41.614820 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:41.615829 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:41.616037 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:41.616126 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:41.618665 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:17:41.618666 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:17:41.618666 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:17:41.618673 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:17:41.619949 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:41.620348 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:41.620349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:41.620350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:41.620410 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:41.620636 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:41.620636 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:41.620636 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:41.620745 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:17:41.620799 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 06:17:41.621077 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 06:17:41.621145 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 06:17:41.621413 16692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:41.612260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.195:56319)
0926 06:17:41.612446 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:41.612483 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 06:17:41.613055 (+ 572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:17:41.613282 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:17:41.613286 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:17:41.613947 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:17:41.614147 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:41.614151 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:41.614906 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:41.614906 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:41.615703 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:41.615705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:41.616139 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:41.616140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:41.616273 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:41.616881 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:17:41.616890 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:17:41.618583 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:17:41.620020 (+ 1437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:17:41.620025 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:17:41.620027 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:17:41.620262 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:17:41.620482 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:17:41.620482 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:17:41.620483 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:17:41.620578 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:17:41.620815 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:17:41.620819 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:17:41.620937 (+ 118us) client_negotiation.cc:769] Sending connection context
0926 06:17:41.621027 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 06:17:41.621152 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":106,"threads_started":1}
I20260926 06:17:41.622136 16690 heartbeater.cc:347] Connected to a master server at 127.2.110.254:35291
I20260926 06:17:41.622252 16690 heartbeater.cc:464] Registering TS with master...
I20260926 06:17:41.622408 16690 heartbeater.cc:511] Master 127.2.110.254:35291 requested a full tablet report, sending...
I20260926 06:17:41.622797 16231 ts_manager.cc:194] Registered new tserver with Master: 1c6be47ad34f432987a5d2ef28f0123f (127.2.110.195:33005)
I20260926 06:17:41.623142 16231 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.195@KRBTEST.COM'} at 127.2.110.195:56319
I20260926 06:17:41.630148 2491 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 06:17:41.638206 16270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:41.631553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44288 (local address 127.2.110.254:35291)
0926 06:17:41.631665 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 06:17:41.631668 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:17:41.631776 (+ 108us) server_negotiation.cc:407] Connection header received
0926 06:17:41.631860 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:17:41.631861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:17:41.631872 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:17:41.631943 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:17:41.632556 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:41.633074 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:41.633913 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:41.634112 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:41.634187 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:41.635538 (+ 1351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:17:41.635540 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:17:41.635540 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:17:41.635548 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:17:41.636924 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:41.637312 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:41.637314 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:41.637314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:41.637434 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:41.637643 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:41.637644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:41.637644 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:41.637840 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:17:41.637939 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 06:17:41.638052 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 06:17:41.638134 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 06:17:41.639886 16231 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44288:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:17:41.641738 16231 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:17:41.649271 16703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:41.644887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51056 (local address 127.2.110.194:46487)
0926 06:17:41.645394 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0926 06:17:41.645399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:17:41.645412 (+ 13us) server_negotiation.cc:407] Connection header received
0926 06:17:41.645454 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:17:41.645456 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:17:41.645473 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:17:41.645586 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:17:41.645767 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:41.646903 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:41.648203 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:41.648950 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:41.649038 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:41.649092 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 06:17:41.649170 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 06:17:41.649205 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":122,"threads_started":1}
I20260926 06:17:41.649541 16705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:41.645156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40336 (local address 127.2.110.195:33005)
0926 06:17:41.645779 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0926 06:17:41.645783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:17:41.645916 (+ 133us) server_negotiation.cc:407] Connection header received
0926 06:17:41.645962 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:17:41.645964 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:17:41.645979 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:17:41.646059 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:17:41.646180 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:41.647219 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:41.648167 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:41.648800 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:41.648871 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:41.648932 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 06:17:41.649390 (+ 458us) server_negotiation.cc:299] Negotiation successful
0926 06:17:41.649446 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":83,"threads_started":1}
I20260926 06:17:41.649209 16704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:41.645053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.195:33005 (local address 127.0.0.1:40336)
0926 06:17:41.645834 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:41.645843 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 06:17:41.645900 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:17:41.646074 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:17:41.646074 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:17:41.646076 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:17:41.646120 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:41.646122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:41.647322 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:41.647323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:41.648068 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:41.648070 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:41.648872 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:41.648873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:41.648969 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:41.648983 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 06:17:41.649058 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 06:17:41.649114 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"thread_start_us":93,"threads_started":1}
I20260926 06:17:41.649899 16702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:41.644771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.194:46487 (local address 127.0.0.1:51056)
0926 06:17:41.645249 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:41.645258 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 06:17:41.645297 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:17:41.645598 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:17:41.645602 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:17:41.645604 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:17:41.645677 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:41.645680 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:41.647021 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:41.647022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:41.648102 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:41.648104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:41.649047 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:41.649048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:41.649103 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:41.649107 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:17:41.649160 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 06:17:41.649193 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"mutex_wait_us":48,"thread_start_us":83,"threads_started":1}
I20260926 06:17:41.649915 16701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:41.644771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.193:43911 (local address 127.0.0.1:47508)
0926 06:17:41.645013 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:41.645057 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 06:17:41.645134 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:17:41.645639 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:17:41.645640 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:17:41.645642 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:17:41.645699 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:41.645701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:41.646758 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:41.646759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:41.648099 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:41.648102 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:41.649521 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:41.649522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:41.649589 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:41.649594 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 06:17:41.649653 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 06:17:41.649703 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":3,"thread_start_us":91,"threads_started":1}
I20260926 06:17:41.650595 16490 tablet_service.cc:1467] Processing CreateTablet for tablet 7ab8e44c23e94034934edce82dbeeff5 (DEFAULT_TABLE table=test-table [id=88eb031892084f13a5e4e67de2643a39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:17:41.650599 16624 tablet_service.cc:1467] Processing CreateTablet for tablet 7ab8e44c23e94034934edce82dbeeff5 (DEFAULT_TABLE table=test-table [id=88eb031892084f13a5e4e67de2643a39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:17:41.650955 16624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ab8e44c23e94034934edce82dbeeff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:17:41.651007 16490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ab8e44c23e94034934edce82dbeeff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:17:41.653815 16707 tablet_bootstrap.cc:492] T 7ab8e44c23e94034934edce82dbeeff5 P deaf547998bd4cce8d3970dce1eb51ad: Bootstrap starting.
I20260926 06:17:41.653815 16708 tablet_bootstrap.cc:492] T 7ab8e44c23e94034934edce82dbeeff5 P 1c6be47ad34f432987a5d2ef28f0123f: Bootstrap starting.
I20260926 06:17:41.654232 16706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:41.645235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47508 (local address 127.2.110.193:43911)
0926 06:17:41.645468 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 06:17:41.645492 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0926 06:17:41.645501 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:17:41.645533 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:17:41.645534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:17:41.645546 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:17:41.645639 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:17:41.645789 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:41.646654 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:41.648204 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:41.648967 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:41.649038 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:41.649092 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 06:17:41.649823 (+ 731us) server_negotiation.cc:299] Negotiation successful
0926 06:17:41.654112 (+ 4289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":83,"threads_started":1}
I20260926 06:17:41.654560 16708 tablet_bootstrap.cc:654] T 7ab8e44c23e94034934edce82dbeeff5 P 1c6be47ad34f432987a5d2ef28f0123f: Neither blocks nor log segments found. Creating new log.
I20260926 06:17:41.654565 16707 tablet_bootstrap.cc:654] T 7ab8e44c23e94034934edce82dbeeff5 P deaf547998bd4cce8d3970dce1eb51ad: Neither blocks nor log segments found. Creating new log.
I20260926 06:17:41.655086 16708 log.cc:824] T 7ab8e44c23e94034934edce82dbeeff5 P 1c6be47ad34f432987a5d2ef28f0123f: Log is configured to *not* fsync() on all Append() calls
I20260926 06:17:41.654971 16356 tablet_service.cc:1467] Processing CreateTablet for tablet 7ab8e44c23e94034934edce82dbeeff5 (DEFAULT_TABLE table=test-table [id=88eb031892084f13a5e4e67de2643a39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:17:41.655176 16707 log.cc:824] T 7ab8e44c23e94034934edce82dbeeff5 P deaf547998bd4cce8d3970dce1eb51ad: Log is configured to *not* fsync() on all Append() calls
I20260926 06:17:41.655364 16356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ab8e44c23e94034934edce82dbeeff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:17:41.656046 16708 tablet_bootstrap.cc:492] T 7ab8e44c23e94034934edce82dbeeff5 P 1c6be47ad34f432987a5d2ef28f0123f: No bootstrap required, opened a new log
I20260926 06:17:41.656136 16708 ts_tablet_manager.cc:1397] T 7ab8e44c23e94034934edce82dbeeff5 P 1c6be47ad34f432987a5d2ef28f0123f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:17:41.658191 16708 raft_consensus.cc:348] T 7ab8e44c23e94034934edce82dbeeff5 P 1c6be47ad34f432987a5d2ef28f0123f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deaf547998bd4cce8d3970dce1eb51ad" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 46487 } } peers { permanent_uuid: "8778addf8daf4894b885ea833fac225c" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 43911 } } peers { permanent_uuid: "1c6be47ad34f432987a5d2ef28f0123f" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 33005 } }
I20260926 06:17:41.658399 16708 raft_consensus.cc:374] T 7ab8e44c23e94034934edce82dbeeff5 P 1c6be47ad34f432987a5d2ef28f0123f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:17:41.658439 16708 raft_consensus.cc:729] T 7ab8e44c23e94034934edce82dbeeff5 P 1c6be47ad34f432987a5d2ef28f0123f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c6be47ad34f432987a5d2ef28f0123f, State: Initialized, Role: FOLLOWER
I20260926 06:17:41.658581 16708 consensus_queue.cc:260] T 7ab8e44c23e94034934edce82dbeeff5 P 1c6be47ad34f432987a5d2ef28f0123f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deaf547998bd4cce8d3970dce1eb51ad" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 46487 } } peers { permanent_uuid: "8778addf8daf4894b885ea833fac225c" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 43911 } } peers { permanent_uuid: "1c6be47ad34f432987a5d2ef28f0123f" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 33005 } }
I20260926 06:17:41.658939 16708 ts_tablet_manager.cc:1428] T 7ab8e44c23e94034934edce82dbeeff5 P 1c6be47ad34f432987a5d2ef28f0123f: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 06:17:41.658990 16711 tablet_bootstrap.cc:492] T 7ab8e44c23e94034934edce82dbeeff5 P 8778addf8daf4894b885ea833fac225c: Bootstrap starting.
I20260926 06:17:41.659085 16690 heartbeater.cc:503] Master 127.2.110.254:35291 was elected leader, sending a full tablet report...
I20260926 06:17:41.659578 16707 tablet_bootstrap.cc:492] T 7ab8e44c23e94034934edce82dbeeff5 P deaf547998bd4cce8d3970dce1eb51ad: No bootstrap required, opened a new log
I20260926 06:17:41.659647 16711 tablet_bootstrap.cc:654] T 7ab8e44c23e94034934edce82dbeeff5 P 8778addf8daf4894b885ea833fac225c: Neither blocks nor log segments found. Creating new log.
I20260926 06:17:41.659668 16707 ts_tablet_manager.cc:1397] T 7ab8e44c23e94034934edce82dbeeff5 P deaf547998bd4cce8d3970dce1eb51ad: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260926 06:17:41.660290 16711 log.cc:824] T 7ab8e44c23e94034934edce82dbeeff5 P 8778addf8daf4894b885ea833fac225c: Log is configured to *not* fsync() on all Append() calls
I20260926 06:17:41.661406 16711 tablet_bootstrap.cc:492] T 7ab8e44c23e94034934edce82dbeeff5 P 8778addf8daf4894b885ea833fac225c: No bootstrap required, opened a new log
I20260926 06:17:41.661481 16711 ts_tablet_manager.cc:1397] T 7ab8e44c23e94034934edce82dbeeff5 P 8778addf8daf4894b885ea833fac225c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 06:17:41.661840 16707 raft_consensus.cc:348] T 7ab8e44c23e94034934edce82dbeeff5 P deaf547998bd4cce8d3970dce1eb51ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deaf547998bd4cce8d3970dce1eb51ad" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 46487 } } peers { permanent_uuid: "8778addf8daf4894b885ea833fac225c" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 43911 } } peers { permanent_uuid: "1c6be47ad34f432987a5d2ef28f0123f" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 33005 } }
I20260926 06:17:41.662026 16707 raft_consensus.cc:374] T 7ab8e44c23e94034934edce82dbeeff5 P deaf547998bd4cce8d3970dce1eb51ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:17:41.662065 16707 raft_consensus.cc:729] T 7ab8e44c23e94034934edce82dbeeff5 P deaf547998bd4cce8d3970dce1eb51ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: deaf547998bd4cce8d3970dce1eb51ad, State: Initialized, Role: FOLLOWER
I20260926 06:17:41.662175 16707 consensus_queue.cc:260] T 7ab8e44c23e94034934edce82dbeeff5 P deaf547998bd4cce8d3970dce1eb51ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deaf547998bd4cce8d3970dce1eb51ad" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 46487 } } peers { permanent_uuid: "8778addf8daf4894b885ea833fac225c" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 43911 } } peers { permanent_uuid: "1c6be47ad34f432987a5d2ef28f0123f" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 33005 } }
I20260926 06:17:41.662479 16707 ts_tablet_manager.cc:1428] T 7ab8e44c23e94034934edce82dbeeff5 P deaf547998bd4cce8d3970dce1eb51ad: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 06:17:41.662495 16556 heartbeater.cc:503] Master 127.2.110.254:35291 was elected leader, sending a full tablet report...
I20260926 06:17:41.663391 16711 raft_consensus.cc:348] T 7ab8e44c23e94034934edce82dbeeff5 P 8778addf8daf4894b885ea833fac225c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deaf547998bd4cce8d3970dce1eb51ad" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 46487 } } peers { permanent_uuid: "8778addf8daf4894b885ea833fac225c" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 43911 } } peers { permanent_uuid: "1c6be47ad34f432987a5d2ef28f0123f" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 33005 } }
I20260926 06:17:41.663583 16711 raft_consensus.cc:374] T 7ab8e44c23e94034934edce82dbeeff5 P 8778addf8daf4894b885ea833fac225c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:17:41.663635 16711 raft_consensus.cc:729] T 7ab8e44c23e94034934edce82dbeeff5 P 8778addf8daf4894b885ea833fac225c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8778addf8daf4894b885ea833fac225c, State: Initialized, Role: FOLLOWER
I20260926 06:17:41.663774 16711 consensus_queue.cc:260] T 7ab8e44c23e94034934edce82dbeeff5 P 8778addf8daf4894b885ea833fac225c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deaf547998bd4cce8d3970dce1eb51ad" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 46487 } } peers { permanent_uuid: "8778addf8daf4894b885ea833fac225c" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 43911 } } peers { permanent_uuid: "1c6be47ad34f432987a5d2ef28f0123f" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 33005 } }
I20260926 06:17:41.664067 16711 ts_tablet_manager.cc:1428] T 7ab8e44c23e94034934edce82dbeeff5 P 8778addf8daf4894b885ea833fac225c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 06:17:41.664096 16422 heartbeater.cc:503] Master 127.2.110.254:35291 was elected leader, sending a full tablet report...
W20260926 06:17:41.686235 16557 tablet.cc:2369] T 7ab8e44c23e94034934edce82dbeeff5 P deaf547998bd4cce8d3970dce1eb51ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:17:41.741449 16712 raft_consensus.cc:482] T 7ab8e44c23e94034934edce82dbeeff5 P 1c6be47ad34f432987a5d2ef28f0123f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:17:41.741636 16712 raft_consensus.cc:504] T 7ab8e44c23e94034934edce82dbeeff5 P 1c6be47ad34f432987a5d2ef28f0123f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deaf547998bd4cce8d3970dce1eb51ad" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 46487 } } peers { permanent_uuid: "8778addf8daf4894b885ea833fac225c" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 43911 } } peers { permanent_uuid: "1c6be47ad34f432987a5d2ef28f0123f" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 33005 } }
I20260926 06:17:41.742090 16712 leader_election.cc:288] T 7ab8e44c23e94034934edce82dbeeff5 P 1c6be47ad34f432987a5d2ef28f0123f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers deaf547998bd4cce8d3970dce1eb51ad (127.2.110.194:46487), 8778addf8daf4894b885ea833fac225c (127.2.110.193:43911)
I20260926 06:17:41.746580 16703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:41.742388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:60577 (local address 127.2.110.194:46487)
0926 06:17:41.742439 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 06:17:41.742441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:17:41.742452 (+ 11us) server_negotiation.cc:407] Connection header received
0926 06:17:41.742548 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:17:41.742550 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:17:41.742557 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:17:41.742642 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:17:41.742840 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:41.743612 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:41.745287 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:41.746105 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:41.746213 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:41.746224 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 06:17:41.746452 (+ 228us) server_negotiation.cc:299] Negotiation successful
0926 06:17:41.746508 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 06:17:41.746568 16692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:41.742285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.194:46487 (local address 127.2.110.195:60577)
0926 06:17:41.742394 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:41.742408 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 06:17:41.742466 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:17:41.742661 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:17:41.742662 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:17:41.742664 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:17:41.742755 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:41.742758 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:41.743735 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:41.743736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:41.745132 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:41.745136 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:41.746230 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:41.746231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:41.746336 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:41.746337 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 06:17:41.746410 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 06:17:41.746467 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"mutex_wait_us":82}
I20260926 06:17:41.746579 16706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:41.742403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:52025 (local address 127.2.110.193:43911)
0926 06:17:41.742466 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 06:17:41.742468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:17:41.742608 (+ 140us) server_negotiation.cc:407] Connection header received
0926 06:17:41.742686 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:17:41.742687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:17:41.742695 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:17:41.742746 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:17:41.742915 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:41.743629 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:41.745286 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:41.746107 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:41.746206 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:41.746220 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0926 06:17:41.746453 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 06:17:41.746509 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 06:17:41.746578 16716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:41.742347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.193:43911 (local address 127.2.110.195:52025)
0926 06:17:41.742527 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:41.742538 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:17:41.742596 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:17:41.742764 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:17:41.742765 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:17:41.742767 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:17:41.742833 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:41.742835 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:41.743735 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:41.743736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:41.745132 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:41.745136 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:41.746230 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:41.746231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:41.746330 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:41.746336 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 06:17:41.746410 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 06:17:41.746466 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":75,"thread_start_us":112,"threads_started":1}
I20260926 06:17:41.746999 16510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ab8e44c23e94034934edce82dbeeff5" candidate_uuid: "1c6be47ad34f432987a5d2ef28f0123f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "deaf547998bd4cce8d3970dce1eb51ad" is_pre_election: true
I20260926 06:17:41.747000 16376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ab8e44c23e94034934edce82dbeeff5" candidate_uuid: "1c6be47ad34f432987a5d2ef28f0123f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8778addf8daf4894b885ea833fac225c" is_pre_election: true
I20260926 06:17:41.747159 16510 raft_consensus.cc:2454] T 7ab8e44c23e94034934edce82dbeeff5 P deaf547998bd4cce8d3970dce1eb51ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c6be47ad34f432987a5d2ef28f0123f in term 0.
I20260926 06:17:41.747171 16376 raft_consensus.cc:2454] T 7ab8e44c23e94034934edce82dbeeff5 P 8778addf8daf4894b885ea833fac225c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c6be47ad34f432987a5d2ef28f0123f in term 0.
I20260926 06:17:41.747423 16578 leader_election.cc:302] T 7ab8e44c23e94034934edce82dbeeff5 P 1c6be47ad34f432987a5d2ef28f0123f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c6be47ad34f432987a5d2ef28f0123f, deaf547998bd4cce8d3970dce1eb51ad; no voters:
I20260926 06:17:41.747579 16712 raft_consensus.cc:2781] T 7ab8e44c23e94034934edce82dbeeff5 P 1c6be47ad34f432987a5d2ef28f0123f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:17:41.747637 16712 raft_consensus.cc:482] T 7ab8e44c23e94034934edce82dbeeff5 P 1c6be47ad34f432987a5d2ef28f0123f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:17:41.747665 16712 raft_consensus.cc:3037] T 7ab8e44c23e94034934edce82dbeeff5 P 1c6be47ad34f432987a5d2ef28f0123f [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:17:41.748449 16712 raft_consensus.cc:504] T 7ab8e44c23e94034934edce82dbeeff5 P 1c6be47ad34f432987a5d2ef28f0123f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deaf547998bd4cce8d3970dce1eb51ad" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 46487 } } peers { permanent_uuid: "8778addf8daf4894b885ea833fac225c" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 43911 } } peers { permanent_uuid: "1c6be47ad34f432987a5d2ef28f0123f" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 33005 } }
I20260926 06:17:41.748618 16712 leader_election.cc:288] T 7ab8e44c23e94034934edce82dbeeff5 P 1c6be47ad34f432987a5d2ef28f0123f [CANDIDATE]: Term 1 election: Requested vote from peers deaf547998bd4cce8d3970dce1eb51ad (127.2.110.194:46487), 8778addf8daf4894b885ea833fac225c (127.2.110.193:43911)
I20260926 06:17:41.748826 16376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ab8e44c23e94034934edce82dbeeff5" candidate_uuid: "1c6be47ad34f432987a5d2ef28f0123f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8778addf8daf4894b885ea833fac225c"
I20260926 06:17:41.748824 16510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ab8e44c23e94034934edce82dbeeff5" candidate_uuid: "1c6be47ad34f432987a5d2ef28f0123f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "deaf547998bd4cce8d3970dce1eb51ad"
I20260926 06:17:41.748924 16510 raft_consensus.cc:3037] T 7ab8e44c23e94034934edce82dbeeff5 P deaf547998bd4cce8d3970dce1eb51ad [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:17:41.748941 16376 raft_consensus.cc:3037] T 7ab8e44c23e94034934edce82dbeeff5 P 8778addf8daf4894b885ea833fac225c [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:17:41.749804 16510 raft_consensus.cc:2454] T 7ab8e44c23e94034934edce82dbeeff5 P deaf547998bd4cce8d3970dce1eb51ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c6be47ad34f432987a5d2ef28f0123f in term 1.
I20260926 06:17:41.749884 16376 raft_consensus.cc:2454] T 7ab8e44c23e94034934edce82dbeeff5 P 8778addf8daf4894b885ea833fac225c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c6be47ad34f432987a5d2ef28f0123f in term 1.
I20260926 06:17:41.749986 16578 leader_election.cc:302] T 7ab8e44c23e94034934edce82dbeeff5 P 1c6be47ad34f432987a5d2ef28f0123f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c6be47ad34f432987a5d2ef28f0123f, deaf547998bd4cce8d3970dce1eb51ad; no voters:
I20260926 06:17:41.750106 16712 raft_consensus.cc:2781] T 7ab8e44c23e94034934edce82dbeeff5 P 1c6be47ad34f432987a5d2ef28f0123f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:17:41.750284 16712 raft_consensus.cc:686] T 7ab8e44c23e94034934edce82dbeeff5 P 1c6be47ad34f432987a5d2ef28f0123f [term 1 LEADER]: Becoming Leader. State: Replica: 1c6be47ad34f432987a5d2ef28f0123f, State: Running, Role: LEADER
I20260926 06:17:41.750401 16712 consensus_queue.cc:237] T 7ab8e44c23e94034934edce82dbeeff5 P 1c6be47ad34f432987a5d2ef28f0123f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deaf547998bd4cce8d3970dce1eb51ad" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 46487 } } peers { permanent_uuid: "8778addf8daf4894b885ea833fac225c" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 43911 } } peers { permanent_uuid: "1c6be47ad34f432987a5d2ef28f0123f" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 33005 } }
I20260926 06:17:41.751245 16231 catalog_manager.cc:5495] T 7ab8e44c23e94034934edce82dbeeff5 P 1c6be47ad34f432987a5d2ef28f0123f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c6be47ad34f432987a5d2ef28f0123f (127.2.110.195). New cstate: current_term: 1 leader_uuid: "1c6be47ad34f432987a5d2ef28f0123f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deaf547998bd4cce8d3970dce1eb51ad" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 46487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8778addf8daf4894b885ea833fac225c" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 43911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c6be47ad34f432987a5d2ef28f0123f" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 33005 } health_report { overall_health: HEALTHY } } }
I20260926 06:17:41.765539 16705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:41.763023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40350 (local address 127.2.110.195:33005)
0926 06:17:41.763083 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 06:17:41.763085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:17:41.763096 (+ 11us) server_negotiation.cc:407] Connection header received
0926 06:17:41.763148 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:17:41.763149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:17:41.763167 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:17:41.763248 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:17:41.763418 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:41.763904 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:41.764688 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:41.764892 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:41.764969 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:41.765123 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:17:41.765222 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:17:41.765317 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 06:17:41.765399 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 06:17:41.765421 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 06:17:41.767418 16376 raft_consensus.cc:1261] T 7ab8e44c23e94034934edce82dbeeff5 P 8778addf8daf4894b885ea833fac225c [term 1 FOLLOWER]: Refusing update from remote peer 1c6be47ad34f432987a5d2ef28f0123f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:17:41.767474 16510 raft_consensus.cc:1261] T 7ab8e44c23e94034934edce82dbeeff5 P deaf547998bd4cce8d3970dce1eb51ad [term 1 FOLLOWER]: Refusing update from remote peer 1c6be47ad34f432987a5d2ef28f0123f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:17:41.767812 16717 consensus_queue.cc:1035] T 7ab8e44c23e94034934edce82dbeeff5 P 1c6be47ad34f432987a5d2ef28f0123f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8778addf8daf4894b885ea833fac225c" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 43911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:17:41.767962 16712 consensus_queue.cc:1035] T 7ab8e44c23e94034934edce82dbeeff5 P 1c6be47ad34f432987a5d2ef28f0123f [LEADER]: Connected to new peer: Peer: permanent_uuid: "deaf547998bd4cce8d3970dce1eb51ad" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 46487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:17:41.770519 16722 mvcc.cc:205] Tried to move back new op lower bound from 7333492579396407296 to 7333492579330396160. Current Snapshot: MvccSnapshot[applied={T|T < 7333492579396407296}]
I20260926 06:17:41.770954 16721 mvcc.cc:205] Tried to move back new op lower bound from 7333492579396407296 to 7333492579330396160. Current Snapshot: MvccSnapshot[applied={T|T < 7333492579396407296}]
I20260926 06:17:41.773030 16229 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:17:41.773141 16229 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:17:41.774009 16229 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d: Sending DeleteTablet for 3 replicas of tablet 7ab8e44c23e94034934edce82dbeeff5
I20260926 06:17:41.774398 16624 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ab8e44c23e94034934edce82dbeeff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:17:41 UTC) from {username='kudu', principal='kudu/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:40336
I20260926 06:17:41.774425 16490 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ab8e44c23e94034934edce82dbeeff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:17:41 UTC) from {username='kudu', principal='kudu/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:51056
I20260926 06:17:41.774654 16731 tablet_replica.cc:330] T 7ab8e44c23e94034934edce82dbeeff5 P deaf547998bd4cce8d3970dce1eb51ad: stopping tablet replica
I20260926 06:17:41.774658 16356 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ab8e44c23e94034934edce82dbeeff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:17:41 UTC) from {username='kudu', principal='kudu/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:47508
I20260926 06:17:41.774770 16731 raft_consensus.cc:2229] T 7ab8e44c23e94034934edce82dbeeff5 P deaf547998bd4cce8d3970dce1eb51ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:17:41.774881 16731 raft_consensus.cc:2258] T 7ab8e44c23e94034934edce82dbeeff5 P deaf547998bd4cce8d3970dce1eb51ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:17:41.775018 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 16293
I20260926 06:17:41.776302 16731 ts_tablet_manager.cc:1905] T 7ab8e44c23e94034934edce82dbeeff5 P deaf547998bd4cce8d3970dce1eb51ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:17:41.778102 16731 ts_tablet_manager.cc:1918] T 7ab8e44c23e94034934edce82dbeeff5 P deaf547998bd4cce8d3970dce1eb51ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:17:41.778188 16731 log.cc:1196] T 7ab8e44c23e94034934edce82dbeeff5 P deaf547998bd4cce8d3970dce1eb51ad: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-1/wal/wals/7ab8e44c23e94034934edce82dbeeff5
I20260926 06:17:41.778491 16731 ts_tablet_manager.cc:1939] T 7ab8e44c23e94034934edce82dbeeff5 P deaf547998bd4cce8d3970dce1eb51ad: Deleting consensus metadata
I20260926 06:17:41.779088 16216 catalog_manager.cc:4841] TS deaf547998bd4cce8d3970dce1eb51ad (127.2.110.194:46487): tablet 7ab8e44c23e94034934edce82dbeeff5 (table test-table [id=88eb031892084f13a5e4e67de2643a39]) successfully deleted
W20260926 06:17:41.781455 16215 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.110.193:43911 (error 108)
I20260926 06:17:41.782032 16704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:41.781888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.193:43911 (local address 127.0.0.1:47520)
0926 06:17:41.781951 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:41.781992 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.193:43911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 06:17:41.782058 16730 tablet_replica.cc:330] T 7ab8e44c23e94034934edce82dbeeff5 P 1c6be47ad34f432987a5d2ef28f0123f: stopping tablet replica
I20260926 06:17:41.782168 16730 raft_consensus.cc:2229] T 7ab8e44c23e94034934edce82dbeeff5 P 1c6be47ad34f432987a5d2ef28f0123f [term 1 LEADER]: Raft consensus shutting down.
W20260926 06:17:41.782156 16215 catalog_manager.cc:4584] TS 8778addf8daf4894b885ea833fac225c (127.2.110.193:43911): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7ab8e44c23e94034934edce82dbeeff5: Network error: Client connection negotiation failed: client connection to 127.2.110.193:43911: connect: Connection refused (error 111)
I20260926 06:17:41.782284 16730 raft_consensus.cc:2258] T 7ab8e44c23e94034934edce82dbeeff5 P 1c6be47ad34f432987a5d2ef28f0123f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:17:41.782509 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 16427
I20260926 06:17:41.788604 16730 ts_tablet_manager.cc:1905] T 7ab8e44c23e94034934edce82dbeeff5 P 1c6be47ad34f432987a5d2ef28f0123f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:17:41.790230 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 16561
I20260926 06:17:41.794864 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 16199
I20260926 06:17:41.794986 16704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:41.794838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.195:33005 (local address 127.0.0.1:40364)
0926 06:17:41.794901 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:41.794934 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.195:33005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 06:17:41.800472 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:35291
--webserver_interface=127.2.110.254
--webserver_port=43605
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:43301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.110.254:35291
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:17:41.886178 16734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:17:41.886507 16734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:17:41.886576 16734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:17:41.887830 16734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:17:41.887873 16734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:17:41.887888 16734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:17:41.887902 16734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:17:41.887914 16734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:17:41.889452 16734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:43301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.2.110.254:35291
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:35291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.110.254
--webserver_port=43605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:17:41.889675 16734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:17:41.889856 16734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:17:41.898653 16741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:17:41.898864 16734 server_base.cc:1034] running on GCE node
W20260926 06:17:41.898698 16743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:17:41.898645 16740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:17:41.899264 16734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:17:41.899610 16734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:17:41.900794 16734 hybrid_clock.cc:648] HybridClock initialized: now 1790403461900766 us; error 45 us; skew 500 ppm
Sep 26 06:17:41 dist-test-slave-hndd krb5kdc[16186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403461, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:17:41.903674 16734 init.cc:377] Logged in from keytab as kudu/127.2.110.254@KRBTEST.COM (short username kudu)
I20260926 06:17:41.904256 16734 webserver.cc:460] Webserver started at http://127.2.110.254:43605/ using document root <none> and password file <none>
I20260926 06:17:41.904528 16734 fs_manager.cc:359] Metadata directory not provided
I20260926 06:17:41.904630 16734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:17:41.906200 16734 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:17:41.906970 16750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:17:41.907171 16734 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:17:41.907303 16734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "a58d836c12134ed295238497e9f8f94d"
format_stamp: "Formatted at 2026-09-26 06:17:41 on dist-test-slave-hndd"
server_key: "a7204bff9c35a98b89c459ba365ae40a"
server_key_iv: "6cc563169f4155333caa39a55ad03acf"
server_key_version: "encryptionkey@0"
I20260926 06:17:41.907776 16734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:17:41.915318 16734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:17:41.917426 16734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:17:41.917591 16734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:17:41.921772 16734 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.254:35291
I20260926 06:17:41.921810 16802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.254:35291 every 8 connection(s)
I20260926 06:17:41.922201 16734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:17:41.923804 16803 sys_catalog.cc:263] Verifying existing consensus state
I20260926 06:17:41.924487 16803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d: Bootstrap starting.
I20260926 06:17:41.925796 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 16734
I20260926 06:17:41.926087 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.193:43911
--local_ip_for_outbound_sockets=127.2.110.193
--tserver_master_addrs=127.2.110.254:35291
--webserver_port=44895
--webserver_interface=127.2.110.193
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab
--principal=kudu/127.2.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:43301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 06:17:41.933652 16803 log.cc:824] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d: Log is configured to *not* fsync() on all Append() calls
I20260926 06:17:41.935564 16803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 06:17:41.935902 16803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d: Bootstrap complete.
I20260926 06:17:41.938304 16803 raft_consensus.cc:348] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58d836c12134ed295238497e9f8f94d" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 35291 } }
I20260926 06:17:41.938535 16803 raft_consensus.cc:729] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a58d836c12134ed295238497e9f8f94d, State: Initialized, Role: FOLLOWER
I20260926 06:17:41.938712 16803 consensus_queue.cc:260] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58d836c12134ed295238497e9f8f94d" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 35291 } }
I20260926 06:17:41.938771 16803 raft_consensus.cc:388] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:17:41.938851 16803 raft_consensus.cc:482] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:17:41.938911 16803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d [term 1 FOLLOWER]: Advancing to term 2
I20260926 06:17:41.940022 16803 raft_consensus.cc:504] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58d836c12134ed295238497e9f8f94d" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 35291 } }
I20260926 06:17:41.940110 16803 leader_election.cc:302] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a58d836c12134ed295238497e9f8f94d; no voters:
I20260926 06:17:41.940342 16803 leader_election.cc:288] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 06:17:41.940376 16807 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d [term 2 FOLLOWER]: Leader election won for term 2
I20260926 06:17:41.940479 16807 raft_consensus.cc:686] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d [term 2 LEADER]: Becoming Leader. State: Replica: a58d836c12134ed295238497e9f8f94d, State: Running, Role: LEADER
I20260926 06:17:41.940582 16807 consensus_queue.cc:237] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58d836c12134ed295238497e9f8f94d" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 35291 } }
I20260926 06:17:41.940618 16803 sys_catalog.cc:564] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:17:41.940858 16808 sys_catalog.cc:455] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a58d836c12134ed295238497e9f8f94d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58d836c12134ed295238497e9f8f94d" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 35291 } } }
I20260926 06:17:41.941133 16808 sys_catalog.cc:458] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d [sys.catalog]: This master's current role is: LEADER
I20260926 06:17:41.941179 16809 sys_catalog.cc:455] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a58d836c12134ed295238497e9f8f94d. Latest consensus state: current_term: 2 leader_uuid: "a58d836c12134ed295238497e9f8f94d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58d836c12134ed295238497e9f8f94d" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 35291 } } }
I20260926 06:17:41.941269 16809 sys_catalog.cc:458] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d [sys.catalog]: This master's current role is: LEADER
I20260926 06:17:41.941488 16817 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:17:41.942272 16817 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:17:41.942412 16817 catalog_manager.cc:1259] Loaded cluster ID: c35f709572054fc4a7c26e4aa20b5718
I20260926 06:17:41.942709 16817 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:17:41.942987 16817 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 06:17:41.947952 16817 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 06:17:41.950639 16817 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:17:41.950841 16817 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d: Loaded TSK: 0
I20260926 06:17:41.951046 16817 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 06:17:42.015877 16805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:17:42.016124 16805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:17:42.016167 16805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:17:42.017418 16805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:17:42.017480 16805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:17:42.017526 16805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.193
I20260926 06:17:42.019042 16805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:43301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab
--principal=kudu/127.2.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.193:43911
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.110.193
--webserver_port=44895
--tserver_master_addrs=127.2.110.254:35291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.193
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:17:42.019562 16805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:17:42.020006 16805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:17:42.026049 16830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:17:42.026063 16831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:17:42.026188 16833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:17:42.026314 16805 server_base.cc:1034] running on GCE node
I20260926 06:17:42.026520 16805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:17:42.026800 16805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:17:42.027992 16805 hybrid_clock.cc:648] HybridClock initialized: now 1790403462027969 us; error 34 us; skew 500 ppm
Sep 26 06:17:42 dist-test-slave-hndd krb5kdc[16186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403462, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:17:42.030855 16805 init.cc:377] Logged in from keytab as kudu/127.2.110.193@KRBTEST.COM (short username kudu)
I20260926 06:17:42.031411 16805 webserver.cc:460] Webserver started at http://127.2.110.193:44895/ using document root <none> and password file <none>
I20260926 06:17:42.031661 16805 fs_manager.cc:359] Metadata directory not provided
I20260926 06:17:42.031719 16805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:17:42.033185 16805 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:17:42.034121 16840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:17:42.034386 16805 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:17:42.034499 16805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-0/wal
uuid: "8778addf8daf4894b885ea833fac225c"
format_stamp: "Formatted at 2026-09-26 06:17:41 on dist-test-slave-hndd"
server_key: "dbe7afcd73ffde686eb3744261deb453"
server_key_iv: "f2ae0f5d2705ce8bfce82f650158c135"
server_key_version: "encryptionkey@0"
I20260926 06:17:42.034907 16805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:17:42.052309 16805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:17:42.054572 16805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:17:42.054747 16805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:17:42.055002 16805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:17:42.055534 16847 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 06:17:42.056986 16805 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 06:17:42.057042 16805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 06:17:42.057068 16805 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 06:17:42.057801 16805 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 06:17:42.057847 16805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 06:17:42.057889 16847 tablet_bootstrap.cc:492] T 7ab8e44c23e94034934edce82dbeeff5 P 8778addf8daf4894b885ea833fac225c: Bootstrap starting.
I20260926 06:17:42.065744 16805 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.193:43911
I20260926 06:17:42.065938 16954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.193:43911 every 8 connection(s)
I20260926 06:17:42.066320 16805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
I20260926 06:17:42.069641 16847 log.cc:824] T 7ab8e44c23e94034934edce82dbeeff5 P 8778addf8daf4894b885ea833fac225c: Log is configured to *not* fsync() on all Append() calls
I20260926 06:17:42.071825 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 16805
I20260926 06:17:42.072139 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.194:46487
--local_ip_for_outbound_sockets=127.2.110.194
--tserver_master_addrs=127.2.110.254:35291
--webserver_port=34939
--webserver_interface=127.2.110.194
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab
--principal=kudu/127.2.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:43301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 06:17:42.073802 16847 tablet_bootstrap.cc:492] T 7ab8e44c23e94034934edce82dbeeff5 P 8778addf8daf4894b885ea833fac225c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 06:17:42.074142 16847 tablet_bootstrap.cc:492] T 7ab8e44c23e94034934edce82dbeeff5 P 8778addf8daf4894b885ea833fac225c: Bootstrap complete.
I20260926 06:17:42.075358 16847 ts_tablet_manager.cc:1397] T 7ab8e44c23e94034934edce82dbeeff5 P 8778addf8daf4894b885ea833fac225c: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.009s
Sep 26 06:17:42 dist-test-slave-hndd krb5kdc[16186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403462, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.193@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:17:42.076853 16847 raft_consensus.cc:348] T 7ab8e44c23e94034934edce82dbeeff5 P 8778addf8daf4894b885ea833fac225c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deaf547998bd4cce8d3970dce1eb51ad" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 46487 } } peers { permanent_uuid: "8778addf8daf4894b885ea833fac225c" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 43911 } } peers { permanent_uuid: "1c6be47ad34f432987a5d2ef28f0123f" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 33005 } }
I20260926 06:17:42.077308 16847 raft_consensus.cc:729] T 7ab8e44c23e94034934edce82dbeeff5 P 8778addf8daf4894b885ea833fac225c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8778addf8daf4894b885ea833fac225c, State: Initialized, Role: FOLLOWER
I20260926 06:17:42.077512 16847 consensus_queue.cc:260] T 7ab8e44c23e94034934edce82dbeeff5 P 8778addf8daf4894b885ea833fac225c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deaf547998bd4cce8d3970dce1eb51ad" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 46487 } } peers { permanent_uuid: "8778addf8daf4894b885ea833fac225c" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 43911 } } peers { permanent_uuid: "1c6be47ad34f432987a5d2ef28f0123f" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 33005 } }
I20260926 06:17:42.077843 16847 ts_tablet_manager.cc:1428] T 7ab8e44c23e94034934edce82dbeeff5 P 8778addf8daf4894b885ea833fac225c: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20260926 06:17:42.080889 16957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:42.067067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.193:45293)
0926 06:17:42.067388 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:42.067438 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 06:17:42.068055 (+ 617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:17:42.070001 (+ 1946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:17:42.070006 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:17:42.070333 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:17:42.070528 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:42.070533 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:42.071673 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:42.071674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:42.072687 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:42.072689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:42.073640 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:42.073641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:42.073788 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:42.074372 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:17:42.074381 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:17:42.076206 (+ 1825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:17:42.078909 (+ 2703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:17:42.078916 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:17:42.078918 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:17:42.079214 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:17:42.079554 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:17:42.079556 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:17:42.079556 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:17:42.079711 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:17:42.080239 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:17:42.080245 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:17:42.080398 (+ 153us) client_negotiation.cc:769] Sending connection context
0926 06:17:42.080517 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 06:17:42.080691 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":129,"threads_started":1}
I20260926 06:17:42.081012 16958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:42.067232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:45293 (local address 127.2.110.254:35291)
0926 06:17:42.069525 (+ 2293us) server_negotiation.cc:206] Beginning negotiation
0926 06:17:42.069535 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 06:17:42.069579 (+ 44us) server_negotiation.cc:407] Connection header received
0926 06:17:42.069712 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:17:42.069718 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:17:42.069771 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:17:42.069924 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:17:42.070662 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:42.071543 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:42.072786 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:42.073424 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:42.073531 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:42.076341 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:17:42.076345 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:17:42.076346 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:17:42.076359 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:17:42.078744 (+ 2385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:42.079315 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:42.079319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:42.079319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:42.079451 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:42.079796 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:42.079797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:42.079797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:42.080136 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:17:42.080220 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 06:17:42.080591 (+ 371us) server_negotiation.cc:299] Negotiation successful
0926 06:17:42.080764 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2241,"thread_start_us":73,"threads_started":1}
I20260926 06:17:42.082105 16955 heartbeater.cc:347] Connected to a master server at 127.2.110.254:35291
I20260926 06:17:42.082226 16955 heartbeater.cc:464] Registering TS with master...
I20260926 06:17:42.082448 16955 heartbeater.cc:511] Master 127.2.110.254:35291 requested a full tablet report, sending...
I20260926 06:17:42.083339 16767 ts_manager.cc:194] Registered new tserver with Master: 8778addf8daf4894b885ea833fac225c (127.2.110.193:43911)
I20260926 06:17:42.084514 16767 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.193@KRBTEST.COM'} at 127.2.110.193:45293
I20260926 06:17:42.085724 16955 heartbeater.cc:503] Master 127.2.110.254:35291 was elected leader, sending a full tablet report...
Sep 26 06:17:42 dist-test-slave-hndd krb5kdc[16186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403461, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.254@KRBTEST.COM for kudu/127.2.110.193@KRBTEST.COM
I20260926 06:17:42.094376 16962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:42.084034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.193:43911 (local address 127.0.0.1:47522)
0926 06:17:42.084407 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:42.084445 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 06:17:42.084515 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:17:42.084826 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:17:42.084835 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:17:42.085264 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:17:42.085368 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:42.085372 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:42.086561 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:42.086562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:42.087714 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:42.087719 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:42.088254 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:42.088255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:42.088356 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:42.088975 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:17:42.088982 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:17:42.090684 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:17:42.093069 (+ 2385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:17:42.093075 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:17:42.093078 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:17:42.093308 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:17:42.093625 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:17:42.093626 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:17:42.093626 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:17:42.093693 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:17:42.094053 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:17:42.094058 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:17:42.094108 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 06:17:42.094173 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 06:17:42.094302 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":126,"threads_started":1}
I20260926 06:17:42.094489 16963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:42.084362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47522 (local address 127.2.110.193:43911)
0926 06:17:42.084660 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0926 06:17:42.084664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:17:42.084674 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:17:42.084707 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:17:42.084708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:17:42.084722 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:17:42.084792 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:17:42.085457 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:42.086471 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:42.087850 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:42.088159 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:42.088226 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:42.090782 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:17:42.090791 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:17:42.090791 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:17:42.090800 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:17:42.092964 (+ 2164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:42.093464 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:42.093468 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:42.093468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:42.093535 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:42.093781 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:42.093782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:42.093783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:42.093982 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:17:42.094067 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 06:17:42.094251 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 06:17:42.094360 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":81,"threads_started":1}
I20260926 06:17:42.094842 16889 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ab8e44c23e94034934edce82dbeeff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:17:41 UTC) from {username='kudu', principal='kudu/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:47522
I20260926 06:17:42.095593 16964 tablet_replica.cc:330] stopping tablet replica
I20260926 06:17:42.095723 16964 raft_consensus.cc:2229] T 7ab8e44c23e94034934edce82dbeeff5 P 8778addf8daf4894b885ea833fac225c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:17:42.095810 16964 raft_consensus.cc:2258] T 7ab8e44c23e94034934edce82dbeeff5 P 8778addf8daf4894b885ea833fac225c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:17:42.096251 16964 ts_tablet_manager.cc:1905] T 7ab8e44c23e94034934edce82dbeeff5 P 8778addf8daf4894b885ea833fac225c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:17:42.098594 16964 ts_tablet_manager.cc:1918] T 7ab8e44c23e94034934edce82dbeeff5 P 8778addf8daf4894b885ea833fac225c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:17:42.098711 16964 log.cc:1196] T 7ab8e44c23e94034934edce82dbeeff5 P 8778addf8daf4894b885ea833fac225c: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-0/wal/wals/7ab8e44c23e94034934edce82dbeeff5
I20260926 06:17:42.098985 16964 ts_tablet_manager.cc:1939] T 7ab8e44c23e94034934edce82dbeeff5 P 8778addf8daf4894b885ea833fac225c: Deleting consensus metadata
I20260926 06:17:42.099642 16751 catalog_manager.cc:4841] TS 8778addf8daf4894b885ea833fac225c (127.2.110.193:43911): tablet 7ab8e44c23e94034934edce82dbeeff5 (table test-table [id=88eb031892084f13a5e4e67de2643a39]) successfully deleted
W20260926 06:17:42.169698 16960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:17:42.169921 16960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:17:42.169947 16960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:17:42.171186 16960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:17:42.171234 16960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:17:42.171263 16960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.194
I20260926 06:17:42.172799 16960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:43301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab
--principal=kudu/127.2.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.194:46487
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.110.194
--webserver_port=34939
--tserver_master_addrs=127.2.110.254:35291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.194
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:17:42.173058 16960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:17:42.173328 16960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:17:42.178669 16970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:17:42.178694 16972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:17:42.178669 16969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:17:42.178730 16960 server_base.cc:1034] running on GCE node
I20260926 06:17:42.179042 16960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:17:42.179307 16960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:17:42.180498 16960 hybrid_clock.cc:648] HybridClock initialized: now 1790403462180495 us; error 40 us; skew 500 ppm
Sep 26 06:17:42 dist-test-slave-hndd krb5kdc[16186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403462, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:17:42.183094 16960 init.cc:377] Logged in from keytab as kudu/127.2.110.194@KRBTEST.COM (short username kudu)
I20260926 06:17:42.183605 16960 webserver.cc:460] Webserver started at http://127.2.110.194:34939/ using document root <none> and password file <none>
I20260926 06:17:42.183852 16960 fs_manager.cc:359] Metadata directory not provided
I20260926 06:17:42.183912 16960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:17:42.185326 16960 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:17:42.186043 16979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:17:42.186304 16960 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:17:42.186380 16960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-1/wal
uuid: "deaf547998bd4cce8d3970dce1eb51ad"
format_stamp: "Formatted at 2026-09-26 06:17:41 on dist-test-slave-hndd"
server_key: "c565334f1559a7d546f2685326074869"
server_key_iv: "e029a9d7333aa4497ca7145f386b377f"
server_key_version: "encryptionkey@0"
I20260926 06:17:42.186756 16960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:17:42.201089 16960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:17:42.203800 16960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:17:42.204037 16960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:17:42.204319 16960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:17:42.204653 16960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:17:42.204710 16960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:17:42.204762 16960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:17:42.204789 16960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:17:42.211059 16960 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.194:46487
I20260926 06:17:42.211126 17092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.194:46487 every 8 connection(s)
I20260926 06:17:42.211505 16960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
I20260926 06:17:42.213523 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 16960
I20260926 06:17:42.213884 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.195:33005
--local_ip_for_outbound_sockets=127.2.110.195
--tserver_master_addrs=127.2.110.254:35291
--webserver_port=38621
--webserver_interface=127.2.110.195
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab
--principal=kudu/127.2.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:43301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 06:17:42 dist-test-slave-hndd krb5kdc[16186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403462, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.194@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:17:42.223068 16958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:42.212234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:32935 (local address 127.2.110.254:35291)
0926 06:17:42.212288 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 06:17:42.212290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:17:42.212512 (+ 222us) server_negotiation.cc:407] Connection header received
0926 06:17:42.213453 (+ 941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:17:42.213455 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:17:42.213470 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:17:42.213545 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:17:42.214740 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:42.215525 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:42.217069 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:42.217277 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:42.217392 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:42.220072 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:17:42.220074 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:17:42.220074 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:17:42.220083 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:17:42.221632 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:42.222065 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:42.222066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:42.222067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:42.222126 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:42.222382 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:42.222383 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:42.222383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:42.222512 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:17:42.222587 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 06:17:42.222878 (+ 291us) server_negotiation.cc:299] Negotiation successful
0926 06:17:42.222968 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 06:17:42.223286 17095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:42.212214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.194:32935)
0926 06:17:42.212378 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:42.212421 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 06:17:42.213201 (+ 780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:17:42.213720 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:17:42.213726 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:17:42.214412 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:17:42.214624 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:42.214629 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:42.215653 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:42.215654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:42.216959 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:42.216963 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:42.217421 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:42.217423 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:42.217594 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:42.218182 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:17:42.218195 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:17:42.219953 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:17:42.221730 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:17:42.221736 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:17:42.221738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:17:42.221983 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:17:42.222198 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:17:42.222199 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:17:42.222199 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:17:42.222307 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:17:42.222590 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:17:42.222597 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:17:42.222738 (+ 141us) client_negotiation.cc:769] Sending connection context
0926 06:17:42.222847 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 06:17:42.223032 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":94,"threads_started":1}
I20260926 06:17:42.223814 17093 heartbeater.cc:347] Connected to a master server at 127.2.110.254:35291
I20260926 06:17:42.223910 17093 heartbeater.cc:464] Registering TS with master...
I20260926 06:17:42.224125 17093 heartbeater.cc:511] Master 127.2.110.254:35291 requested a full tablet report, sending...
I20260926 06:17:42.224524 16767 ts_manager.cc:194] Registered new tserver with Master: deaf547998bd4cce8d3970dce1eb51ad (127.2.110.194:46487)
I20260926 06:17:42.224895 16767 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.194@KRBTEST.COM'} at 127.2.110.194:32935
W20260926 06:17:42.301577 17096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:17:42.301823 17096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:17:42.301863 17096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:17:42.303099 17096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:17:42.303169 17096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:17:42.303220 17096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.195
I20260926 06:17:42.304739 17096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:43301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab
--principal=kudu/127.2.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.195:33005
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.110.195
--webserver_port=38621
--tserver_master_addrs=127.2.110.254:35291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.195
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:17:42.305038 17096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:17:42.305279 17096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:17:42.310261 17102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:17:42.310236 17104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:17:42.310258 17101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:17:42.310678 17096 server_base.cc:1034] running on GCE node
I20260926 06:17:42.310943 17096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:17:42.311192 17096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:17:42.312384 17096 hybrid_clock.cc:648] HybridClock initialized: now 1790403462312384 us; error 42 us; skew 500 ppm
Sep 26 06:17:42 dist-test-slave-hndd krb5kdc[16186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403462, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:17:42.315090 17096 init.cc:377] Logged in from keytab as kudu/127.2.110.195@KRBTEST.COM (short username kudu)
I20260926 06:17:42.315567 17096 webserver.cc:460] Webserver started at http://127.2.110.195:38621/ using document root <none> and password file <none>
I20260926 06:17:42.315819 17096 fs_manager.cc:359] Metadata directory not provided
I20260926 06:17:42.315907 17096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:17:42.317291 17096 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:17:42.318084 17111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:17:42.318379 17096 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:17:42.318493 17096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-2/wal
uuid: "1c6be47ad34f432987a5d2ef28f0123f"
format_stamp: "Formatted at 2026-09-26 06:17:41 on dist-test-slave-hndd"
server_key: "62639a18bf85d59a91212d35060eb7a6"
server_key_iv: "08e8d15d0ea5c3388dbc3b4800c2ef5d"
server_key_version: "encryptionkey@0"
I20260926 06:17:42.318882 17096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:17:42.338385 17096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:17:42.341022 17096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:17:42.341180 17096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:17:42.341480 17096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:17:42.342129 17118 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260926 06:17:42.344115 17118 ts_tablet_manager.cc:1852] T 7ab8e44c23e94034934edce82dbeeff5 P 1c6be47ad34f432987a5d2ef28f0123f: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260926 06:17:42.344638 17118 ts_tablet_manager.cc:1905] T 7ab8e44c23e94034934edce82dbeeff5 P 1c6be47ad34f432987a5d2ef28f0123f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:17:42.347383 17118 ts_tablet_manager.cc:1918] T 7ab8e44c23e94034934edce82dbeeff5 P 1c6be47ad34f432987a5d2ef28f0123f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:17:42.347527 17118 log.cc:1196] T 7ab8e44c23e94034934edce82dbeeff5 P 1c6be47ad34f432987a5d2ef28f0123f: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-2/wal/wals/7ab8e44c23e94034934edce82dbeeff5
I20260926 06:17:42.347831 17118 ts_tablet_manager.cc:1939] T 7ab8e44c23e94034934edce82dbeeff5 P 1c6be47ad34f432987a5d2ef28f0123f: Deleting consensus metadata
I20260926 06:17:42.348078 17096 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260926 06:17:42.348132 17096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260926 06:17:42.348160 17096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:17:42.348173 17096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:17:42.354440 17096 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.195:33005
I20260926 06:17:42.354545 17225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.195:33005 every 8 connection(s)
I20260926 06:17:42.354873 17096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
I20260926 06:17:42.360042 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 17096
Sep 26 06:17:42 dist-test-slave-hndd krb5kdc[16186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403462, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.195@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:17:42.365262 16958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:42.355752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:49357 (local address 127.2.110.254:35291)
0926 06:17:42.355843 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 06:17:42.355846 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:17:42.356060 (+ 214us) server_negotiation.cc:407] Connection header received
0926 06:17:42.356584 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:17:42.356585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:17:42.356598 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:17:42.356652 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:17:42.357287 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:42.357815 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:42.358845 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:42.359059 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:42.359122 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:42.361915 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:17:42.361917 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:17:42.361917 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:17:42.361925 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:17:42.363584 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:42.364109 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:42.364110 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:42.364111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:42.364168 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:42.364501 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:42.364502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:42.364502 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:42.364606 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:17:42.364658 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 06:17:42.365043 (+ 385us) server_negotiation.cc:299] Negotiation successful
0926 06:17:42.365141 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 06:17:42.365375 17228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:42.355711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.195:49357)
0926 06:17:42.355935 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:42.355972 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 06:17:42.356492 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:17:42.356727 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:17:42.356730 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:17:42.356993 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:17:42.357186 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:42.357189 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:42.357927 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:42.357928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:42.358749 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:42.358752 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:42.359160 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:42.359161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:42.359279 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:42.359905 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:17:42.359913 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:17:42.361827 (+ 1914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:17:42.363669 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:17:42.363678 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:17:42.363681 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:17:42.364037 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:17:42.364261 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:17:42.364262 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:17:42.364262 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:17:42.364428 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:17:42.364678 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:17:42.364685 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:17:42.364871 (+ 186us) client_negotiation.cc:769] Sending connection context
0926 06:17:42.364995 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 06:17:42.365137 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":126,"threads_started":1}
I20260926 06:17:42.366026 17226 heartbeater.cc:347] Connected to a master server at 127.2.110.254:35291
I20260926 06:17:42.366135 17226 heartbeater.cc:464] Registering TS with master...
I20260926 06:17:42.366323 17226 heartbeater.cc:511] Master 127.2.110.254:35291 requested a full tablet report, sending...
I20260926 06:17:42.366802 16767 ts_manager.cc:194] Registered new tserver with Master: 1c6be47ad34f432987a5d2ef28f0123f (127.2.110.195:33005)
I20260926 06:17:42.367213 16767 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.195@KRBTEST.COM'} at 127.2.110.195:49357
I20260926 06:17:42.368079 17230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:42.360525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44292 (local address 127.2.110.254:35291)
0926 06:17:42.360678 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 06:17:42.360680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:17:42.360691 (+ 11us) server_negotiation.cc:407] Connection header received
0926 06:17:42.360754 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:17:42.360755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:17:42.360764 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:17:42.360824 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:17:42.361558 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:42.362181 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:42.363162 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:42.363484 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:42.363561 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:42.365020 (+ 1459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:17:42.365022 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:17:42.365022 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:17:42.365028 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:17:42.366816 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:42.367208 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:42.367209 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:42.367209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:42.367316 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:42.367545 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:42.367545 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:42.367546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:42.367677 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:17:42.367844 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 06:17:42.367886 (+ 42us) server_negotiation.cc:299] Negotiation successful
0926 06:17:42.367979 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":80,"threads_started":1}
I20260926 06:17:42.368801 2491 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 06:17:42.371483 17230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:42.369125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44306 (local address 127.2.110.254:35291)
0926 06:17:42.369192 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 06:17:42.369194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:17:42.369299 (+ 105us) server_negotiation.cc:407] Connection header received
0926 06:17:42.369385 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:17:42.369386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:17:42.369420 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:17:42.369488 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:17:42.369586 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:42.370038 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:42.370771 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:42.370955 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:42.371003 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:42.371150 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:17:42.371235 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:17:42.371294 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 06:17:42.371366 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 06:17:42.371383 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 06:17:42.371786 16767 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:44306:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:17:42.373425 16767 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:17:42.375765 16889 tablet_service.cc:1467] Processing CreateTablet for tablet ee565d95b1b5491bb2765d001fcc0cd2 (DEFAULT_TABLE table=test-table [id=301fda98571345e0a0129e7d4145dad4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:17:42.375980 16889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee565d95b1b5491bb2765d001fcc0cd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:17:42.377842 16847 tablet_bootstrap.cc:492] T ee565d95b1b5491bb2765d001fcc0cd2 P 8778addf8daf4894b885ea833fac225c: Bootstrap starting.
I20260926 06:17:42.378244 16847 tablet_bootstrap.cc:654] T ee565d95b1b5491bb2765d001fcc0cd2 P 8778addf8daf4894b885ea833fac225c: Neither blocks nor log segments found. Creating new log.
I20260926 06:17:42.379156 16847 tablet_bootstrap.cc:492] T ee565d95b1b5491bb2765d001fcc0cd2 P 8778addf8daf4894b885ea833fac225c: No bootstrap required, opened a new log
I20260926 06:17:42.379213 16847 ts_tablet_manager.cc:1397] T ee565d95b1b5491bb2765d001fcc0cd2 P 8778addf8daf4894b885ea833fac225c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 06:17:42.379360 16847 raft_consensus.cc:348] T ee565d95b1b5491bb2765d001fcc0cd2 P 8778addf8daf4894b885ea833fac225c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c6be47ad34f432987a5d2ef28f0123f" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 33005 } } peers { permanent_uuid: "deaf547998bd4cce8d3970dce1eb51ad" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 46487 } } peers { permanent_uuid: "8778addf8daf4894b885ea833fac225c" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 43911 } }
I20260926 06:17:42.379439 16847 raft_consensus.cc:374] T ee565d95b1b5491bb2765d001fcc0cd2 P 8778addf8daf4894b885ea833fac225c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:17:42.379492 16847 raft_consensus.cc:729] T ee565d95b1b5491bb2765d001fcc0cd2 P 8778addf8daf4894b885ea833fac225c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8778addf8daf4894b885ea833fac225c, State: Initialized, Role: FOLLOWER
I20260926 06:17:42.379547 16847 consensus_queue.cc:260] T ee565d95b1b5491bb2765d001fcc0cd2 P 8778addf8daf4894b885ea833fac225c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c6be47ad34f432987a5d2ef28f0123f" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 33005 } } peers { permanent_uuid: "deaf547998bd4cce8d3970dce1eb51ad" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 46487 } } peers { permanent_uuid: "8778addf8daf4894b885ea833fac225c" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 43911 } }
I20260926 06:17:42.379637 16847 ts_tablet_manager.cc:1428] T ee565d95b1b5491bb2765d001fcc0cd2 P 8778addf8daf4894b885ea833fac225c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 06:17:42.380137 17232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:42.375444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.194:46487 (local address 127.0.0.1:51068)
0926 06:17:42.375619 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:42.375629 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:17:42.375680 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:17:42.376068 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:17:42.376069 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:17:42.376071 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:17:42.376126 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:42.376128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:42.377499 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:42.377500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:42.378842 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:42.378845 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:42.379857 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:42.379858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:42.379933 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:42.379955 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 06:17:42.380011 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 06:17:42.380060 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":128,"threads_started":1}
I20260926 06:17:42.380157 17234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:42.375699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51068 (local address 127.2.110.194:46487)
0926 06:17:42.375871 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 06:17:42.375876 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:17:42.375895 (+ 19us) server_negotiation.cc:407] Connection header received
0926 06:17:42.375935 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:17:42.375937 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:17:42.375953 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:17:42.376045 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:17:42.376210 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:42.377393 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:42.378948 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:42.379749 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:42.379844 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:42.379907 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 06:17:42.380043 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 06:17:42.380082 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":92,"threads_started":1}
I20260926 06:17:42.380868 17027 tablet_service.cc:1467] Processing CreateTablet for tablet ee565d95b1b5491bb2765d001fcc0cd2 (DEFAULT_TABLE table=test-table [id=301fda98571345e0a0129e7d4145dad4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:17:42.381211 17027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee565d95b1b5491bb2765d001fcc0cd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:17:42.381621 16962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:42.375361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.195:33005 (local address 127.0.0.1:40374)
0926 06:17:42.375420 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:42.375432 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:17:42.375471 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:17:42.377725 (+ 2254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:17:42.377726 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:17:42.377728 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:17:42.377774 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:42.377775 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:42.379106 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:42.379107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:42.380161 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:42.380164 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:42.381321 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:42.381323 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:42.381431 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:42.381436 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 06:17:42.381496 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 06:17:42.381548 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 06:17:42.381637 17233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:42.375504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40374 (local address 127.2.110.195:33005)
0926 06:17:42.377497 (+ 1993us) server_negotiation.cc:206] Beginning negotiation
0926 06:17:42.377502 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:17:42.377512 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:17:42.377549 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:17:42.377551 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:17:42.377568 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:17:42.377690 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:17:42.377906 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:42.379009 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:42.380236 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:42.381215 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:42.381320 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:42.381411 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 06:17:42.381510 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 06:17:42.381548 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1952,"thread_start_us":109,"threads_started":1}
I20260926 06:17:42.382190 17160 tablet_service.cc:1467] Processing CreateTablet for tablet ee565d95b1b5491bb2765d001fcc0cd2 (DEFAULT_TABLE table=test-table [id=301fda98571345e0a0129e7d4145dad4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:17:42.382509 17160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee565d95b1b5491bb2765d001fcc0cd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:17:42.384691 17118 tablet_bootstrap.cc:492] T ee565d95b1b5491bb2765d001fcc0cd2 P 1c6be47ad34f432987a5d2ef28f0123f: Bootstrap starting.
I20260926 06:17:42.384992 17236 tablet_bootstrap.cc:492] T ee565d95b1b5491bb2765d001fcc0cd2 P deaf547998bd4cce8d3970dce1eb51ad: Bootstrap starting.
I20260926 06:17:42.385638 17118 tablet_bootstrap.cc:654] T ee565d95b1b5491bb2765d001fcc0cd2 P 1c6be47ad34f432987a5d2ef28f0123f: Neither blocks nor log segments found. Creating new log.
I20260926 06:17:42.385643 17236 tablet_bootstrap.cc:654] T ee565d95b1b5491bb2765d001fcc0cd2 P deaf547998bd4cce8d3970dce1eb51ad: Neither blocks nor log segments found. Creating new log.
I20260926 06:17:42.385999 17118 log.cc:824] T ee565d95b1b5491bb2765d001fcc0cd2 P 1c6be47ad34f432987a5d2ef28f0123f: Log is configured to *not* fsync() on all Append() calls
I20260926 06:17:42.386296 17236 log.cc:824] T ee565d95b1b5491bb2765d001fcc0cd2 P deaf547998bd4cce8d3970dce1eb51ad: Log is configured to *not* fsync() on all Append() calls
I20260926 06:17:42.386862 17118 tablet_bootstrap.cc:492] T ee565d95b1b5491bb2765d001fcc0cd2 P 1c6be47ad34f432987a5d2ef28f0123f: No bootstrap required, opened a new log
I20260926 06:17:42.386924 17118 ts_tablet_manager.cc:1397] T ee565d95b1b5491bb2765d001fcc0cd2 P 1c6be47ad34f432987a5d2ef28f0123f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:17:42.387006 17236 tablet_bootstrap.cc:492] T ee565d95b1b5491bb2765d001fcc0cd2 P deaf547998bd4cce8d3970dce1eb51ad: No bootstrap required, opened a new log
I20260926 06:17:42.387076 17236 ts_tablet_manager.cc:1397] T ee565d95b1b5491bb2765d001fcc0cd2 P deaf547998bd4cce8d3970dce1eb51ad: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 06:17:42.388331 17118 raft_consensus.cc:348] T ee565d95b1b5491bb2765d001fcc0cd2 P 1c6be47ad34f432987a5d2ef28f0123f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c6be47ad34f432987a5d2ef28f0123f" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 33005 } } peers { permanent_uuid: "deaf547998bd4cce8d3970dce1eb51ad" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 46487 } } peers { permanent_uuid: "8778addf8daf4894b885ea833fac225c" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 43911 } }
I20260926 06:17:42.388468 17118 raft_consensus.cc:374] T ee565d95b1b5491bb2765d001fcc0cd2 P 1c6be47ad34f432987a5d2ef28f0123f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:17:42.388491 17118 raft_consensus.cc:729] T ee565d95b1b5491bb2765d001fcc0cd2 P 1c6be47ad34f432987a5d2ef28f0123f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c6be47ad34f432987a5d2ef28f0123f, State: Initialized, Role: FOLLOWER
I20260926 06:17:42.388392 17236 raft_consensus.cc:348] T ee565d95b1b5491bb2765d001fcc0cd2 P deaf547998bd4cce8d3970dce1eb51ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c6be47ad34f432987a5d2ef28f0123f" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 33005 } } peers { permanent_uuid: "deaf547998bd4cce8d3970dce1eb51ad" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 46487 } } peers { permanent_uuid: "8778addf8daf4894b885ea833fac225c" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 43911 } }
I20260926 06:17:42.388624 17118 consensus_queue.cc:260] T ee565d95b1b5491bb2765d001fcc0cd2 P 1c6be47ad34f432987a5d2ef28f0123f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c6be47ad34f432987a5d2ef28f0123f" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 33005 } } peers { permanent_uuid: "deaf547998bd4cce8d3970dce1eb51ad" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 46487 } } peers { permanent_uuid: "8778addf8daf4894b885ea833fac225c" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 43911 } }
I20260926 06:17:42.388584 17236 raft_consensus.cc:374] T ee565d95b1b5491bb2765d001fcc0cd2 P deaf547998bd4cce8d3970dce1eb51ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:17:42.388706 17236 raft_consensus.cc:729] T ee565d95b1b5491bb2765d001fcc0cd2 P deaf547998bd4cce8d3970dce1eb51ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: deaf547998bd4cce8d3970dce1eb51ad, State: Initialized, Role: FOLLOWER
I20260926 06:17:42.388831 17236 consensus_queue.cc:260] T ee565d95b1b5491bb2765d001fcc0cd2 P deaf547998bd4cce8d3970dce1eb51ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c6be47ad34f432987a5d2ef28f0123f" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 33005 } } peers { permanent_uuid: "deaf547998bd4cce8d3970dce1eb51ad" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 46487 } } peers { permanent_uuid: "8778addf8daf4894b885ea833fac225c" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 43911 } }
I20260926 06:17:42.388996 17118 ts_tablet_manager.cc:1428] T ee565d95b1b5491bb2765d001fcc0cd2 P 1c6be47ad34f432987a5d2ef28f0123f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:17:42.389000 17226 heartbeater.cc:503] Master 127.2.110.254:35291 was elected leader, sending a full tablet report...
I20260926 06:17:42.389194 17236 ts_tablet_manager.cc:1428] T ee565d95b1b5491bb2765d001fcc0cd2 P deaf547998bd4cce8d3970dce1eb51ad: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 06:17:42.389380 17093 heartbeater.cc:503] Master 127.2.110.254:35291 was elected leader, sending a full tablet report...
W20260926 06:17:42.462153 17094 tablet.cc:2369] T ee565d95b1b5491bb2765d001fcc0cd2 P deaf547998bd4cce8d3970dce1eb51ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:17:42.554610 17240 raft_consensus.cc:482] T ee565d95b1b5491bb2765d001fcc0cd2 P deaf547998bd4cce8d3970dce1eb51ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:17:42.554749 17240 raft_consensus.cc:504] T ee565d95b1b5491bb2765d001fcc0cd2 P deaf547998bd4cce8d3970dce1eb51ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c6be47ad34f432987a5d2ef28f0123f" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 33005 } } peers { permanent_uuid: "deaf547998bd4cce8d3970dce1eb51ad" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 46487 } } peers { permanent_uuid: "8778addf8daf4894b885ea833fac225c" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 43911 } }
I20260926 06:17:42.555238 17240 leader_election.cc:288] T ee565d95b1b5491bb2765d001fcc0cd2 P deaf547998bd4cce8d3970dce1eb51ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c6be47ad34f432987a5d2ef28f0123f (127.2.110.195:33005), 8778addf8daf4894b885ea833fac225c (127.2.110.193:43911)
I20260926 06:17:42.558598 17095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:42.555463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.195:33005 (local address 127.2.110.194:44897)
0926 06:17:42.555577 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:42.555595 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 06:17:42.555665 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:17:42.555890 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:17:42.555891 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:17:42.555893 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:17:42.555973 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:42.555975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:42.556835 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:42.556836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:42.557622 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:42.557624 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:42.558202 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:42.558288 (+ 86us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:42.558385 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:42.558390 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 06:17:42.558469 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 06:17:42.558516 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"mutex_wait_us":1}
I20260926 06:17:42.558598 17233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:42.555607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:44897 (local address 127.2.110.195:33005)
0926 06:17:42.555660 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 06:17:42.555662 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:17:42.555674 (+ 12us) server_negotiation.cc:407] Connection header received
0926 06:17:42.555768 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:17:42.555770 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:17:42.555787 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:17:42.555850 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:17:42.556064 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:42.556746 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:42.557724 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:42.558110 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:42.558203 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:42.558288 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 06:17:42.558489 (+ 201us) server_negotiation.cc:299] Negotiation successful
0926 06:17:42.558533 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 06:17:42.558987 17241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:42.555618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.193:43911 (local address 127.2.110.194:59621)
0926 06:17:42.555822 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:42.555868 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 06:17:42.555925 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:17:42.556195 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:17:42.556196 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:17:42.556198 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:17:42.556285 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:42.556287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:42.557112 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:42.557113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:42.558092 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:42.558094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:42.558713 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:42.558715 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:42.558791 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:42.558795 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:17:42.558862 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 06:17:42.558903 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":123,"threads_started":1}
I20260926 06:17:42.559012 16963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:42.555679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:59621 (local address 127.2.110.193:43911)
0926 06:17:42.555735 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 06:17:42.555737 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:17:42.555941 (+ 204us) server_negotiation.cc:407] Connection header received
0926 06:17:42.556018 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:17:42.556019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:17:42.556049 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:17:42.556119 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:17:42.556359 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:42.557010 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:42.558201 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:42.558621 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:42.558713 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:42.558789 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 06:17:42.558892 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 06:17:42.558946 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 06:17:42.559080 17180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee565d95b1b5491bb2765d001fcc0cd2" candidate_uuid: "deaf547998bd4cce8d3970dce1eb51ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c6be47ad34f432987a5d2ef28f0123f" is_pre_election: true
I20260926 06:17:42.559303 17180 raft_consensus.cc:2454] T ee565d95b1b5491bb2765d001fcc0cd2 P 1c6be47ad34f432987a5d2ef28f0123f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate deaf547998bd4cce8d3970dce1eb51ad in term 0.
I20260926 06:17:42.559366 16909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee565d95b1b5491bb2765d001fcc0cd2" candidate_uuid: "deaf547998bd4cce8d3970dce1eb51ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8778addf8daf4894b885ea833fac225c" is_pre_election: true
I20260926 06:17:42.559515 16909 raft_consensus.cc:2454] T ee565d95b1b5491bb2765d001fcc0cd2 P 8778addf8daf4894b885ea833fac225c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate deaf547998bd4cce8d3970dce1eb51ad in term 0.
I20260926 06:17:42.559542 16980 leader_election.cc:302] T ee565d95b1b5491bb2765d001fcc0cd2 P deaf547998bd4cce8d3970dce1eb51ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c6be47ad34f432987a5d2ef28f0123f, deaf547998bd4cce8d3970dce1eb51ad; no voters:
I20260926 06:17:42.559685 17240 raft_consensus.cc:2781] T ee565d95b1b5491bb2765d001fcc0cd2 P deaf547998bd4cce8d3970dce1eb51ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:17:42.559760 17240 raft_consensus.cc:482] T ee565d95b1b5491bb2765d001fcc0cd2 P deaf547998bd4cce8d3970dce1eb51ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:17:42.559787 17240 raft_consensus.cc:3037] T ee565d95b1b5491bb2765d001fcc0cd2 P deaf547998bd4cce8d3970dce1eb51ad [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:17:42.560490 17240 raft_consensus.cc:504] T ee565d95b1b5491bb2765d001fcc0cd2 P deaf547998bd4cce8d3970dce1eb51ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c6be47ad34f432987a5d2ef28f0123f" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 33005 } } peers { permanent_uuid: "deaf547998bd4cce8d3970dce1eb51ad" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 46487 } } peers { permanent_uuid: "8778addf8daf4894b885ea833fac225c" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 43911 } }
I20260926 06:17:42.560606 17240 leader_election.cc:288] T ee565d95b1b5491bb2765d001fcc0cd2 P deaf547998bd4cce8d3970dce1eb51ad [CANDIDATE]: Term 1 election: Requested vote from peers 1c6be47ad34f432987a5d2ef28f0123f (127.2.110.195:33005), 8778addf8daf4894b885ea833fac225c (127.2.110.193:43911)
I20260926 06:17:42.560796 17180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee565d95b1b5491bb2765d001fcc0cd2" candidate_uuid: "deaf547998bd4cce8d3970dce1eb51ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c6be47ad34f432987a5d2ef28f0123f"
I20260926 06:17:42.560904 17180 raft_consensus.cc:3037] T ee565d95b1b5491bb2765d001fcc0cd2 P 1c6be47ad34f432987a5d2ef28f0123f [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:17:42.560876 16909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee565d95b1b5491bb2765d001fcc0cd2" candidate_uuid: "deaf547998bd4cce8d3970dce1eb51ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8778addf8daf4894b885ea833fac225c"
I20260926 06:17:42.561070 16909 raft_consensus.cc:3037] T ee565d95b1b5491bb2765d001fcc0cd2 P 8778addf8daf4894b885ea833fac225c [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:17:42.561782 17180 raft_consensus.cc:2454] T ee565d95b1b5491bb2765d001fcc0cd2 P 1c6be47ad34f432987a5d2ef28f0123f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate deaf547998bd4cce8d3970dce1eb51ad in term 1.
I20260926 06:17:42.561826 16909 raft_consensus.cc:2454] T ee565d95b1b5491bb2765d001fcc0cd2 P 8778addf8daf4894b885ea833fac225c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate deaf547998bd4cce8d3970dce1eb51ad in term 1.
I20260926 06:17:42.561986 16980 leader_election.cc:302] T ee565d95b1b5491bb2765d001fcc0cd2 P deaf547998bd4cce8d3970dce1eb51ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8778addf8daf4894b885ea833fac225c, deaf547998bd4cce8d3970dce1eb51ad; no voters:
I20260926 06:17:42.562088 17240 raft_consensus.cc:2781] T ee565d95b1b5491bb2765d001fcc0cd2 P deaf547998bd4cce8d3970dce1eb51ad [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:17:42.562258 17240 raft_consensus.cc:686] T ee565d95b1b5491bb2765d001fcc0cd2 P deaf547998bd4cce8d3970dce1eb51ad [term 1 LEADER]: Becoming Leader. State: Replica: deaf547998bd4cce8d3970dce1eb51ad, State: Running, Role: LEADER
I20260926 06:17:42.562376 17240 consensus_queue.cc:237] T ee565d95b1b5491bb2765d001fcc0cd2 P deaf547998bd4cce8d3970dce1eb51ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c6be47ad34f432987a5d2ef28f0123f" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 33005 } } peers { permanent_uuid: "deaf547998bd4cce8d3970dce1eb51ad" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 46487 } } peers { permanent_uuid: "8778addf8daf4894b885ea833fac225c" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 43911 } }
I20260926 06:17:42.563272 16767 catalog_manager.cc:5495] T ee565d95b1b5491bb2765d001fcc0cd2 P deaf547998bd4cce8d3970dce1eb51ad reported cstate change: term changed from 0 to 1, leader changed from <none> to deaf547998bd4cce8d3970dce1eb51ad (127.2.110.194). New cstate: current_term: 1 leader_uuid: "deaf547998bd4cce8d3970dce1eb51ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c6be47ad34f432987a5d2ef28f0123f" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 33005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "deaf547998bd4cce8d3970dce1eb51ad" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 46487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8778addf8daf4894b885ea833fac225c" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 43911 } health_report { overall_health: UNKNOWN } } }
W20260926 06:17:42.569931 16956 tablet.cc:2369] T ee565d95b1b5491bb2765d001fcc0cd2 P 8778addf8daf4894b885ea833fac225c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 06:17:42.605609 17227 tablet.cc:2369] T ee565d95b1b5491bb2765d001fcc0cd2 P 1c6be47ad34f432987a5d2ef28f0123f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:17:42.606707 17234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:42.604307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51074 (local address 127.2.110.194:46487)
0926 06:17:42.604367 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 06:17:42.604369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:17:42.604386 (+ 17us) server_negotiation.cc:407] Connection header received
0926 06:17:42.604424 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:17:42.604424 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:17:42.604442 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:17:42.604502 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:17:42.604616 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:42.605077 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:42.605984 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:42.606183 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:42.606248 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:42.606367 (+ 119us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:17:42.606464 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:17:42.606523 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 06:17:42.606567 (+ 44us) server_negotiation.cc:299] Negotiation successful
0926 06:17:42.606584 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 06:17:42.608481 16909 raft_consensus.cc:1261] T ee565d95b1b5491bb2765d001fcc0cd2 P 8778addf8daf4894b885ea833fac225c [term 1 FOLLOWER]: Refusing update from remote peer deaf547998bd4cce8d3970dce1eb51ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:17:42.608485 17180 raft_consensus.cc:1261] T ee565d95b1b5491bb2765d001fcc0cd2 P 1c6be47ad34f432987a5d2ef28f0123f [term 1 FOLLOWER]: Refusing update from remote peer deaf547998bd4cce8d3970dce1eb51ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:17:42.608847 17240 consensus_queue.cc:1035] T ee565d95b1b5491bb2765d001fcc0cd2 P deaf547998bd4cce8d3970dce1eb51ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "8778addf8daf4894b885ea833fac225c" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 43911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:17:42.609002 17242 consensus_queue.cc:1035] T ee565d95b1b5491bb2765d001fcc0cd2 P deaf547998bd4cce8d3970dce1eb51ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c6be47ad34f432987a5d2ef28f0123f" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 33005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:17:42.612303 17247 mvcc.cc:205] Tried to move back new op lower bound from 7333492582841696256 to 7333492582656237568. Current Snapshot: MvccSnapshot[applied={T|T < 7333492582841696256}]
I20260926 06:17:42.614032 16766 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:44306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:17:42.614135 16766 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:44306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:17:42.614920 16766 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d: Sending DeleteTablet for 3 replicas of tablet ee565d95b1b5491bb2765d001fcc0cd2
I20260926 06:17:42.615253 17027 tablet_service.cc:1514] Processing DeleteTablet for tablet ee565d95b1b5491bb2765d001fcc0cd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:17:42 UTC) from {username='kudu', principal='kudu/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:51068
I20260926 06:17:42.615343 17160 tablet_service.cc:1514] Processing DeleteTablet for tablet ee565d95b1b5491bb2765d001fcc0cd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:17:42 UTC) from {username='kudu', principal='kudu/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:40374
I20260926 06:17:42.615597 17255 tablet_replica.cc:330] T ee565d95b1b5491bb2765d001fcc0cd2 P 1c6be47ad34f432987a5d2ef28f0123f: stopping tablet replica
I20260926 06:17:42.615622 16889 tablet_service.cc:1514] Processing DeleteTablet for tablet ee565d95b1b5491bb2765d001fcc0cd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:17:42 UTC) from {username='kudu', principal='kudu/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:47522
I20260926 06:17:42.615648 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 16805
I20260926 06:17:42.615706 17255 raft_consensus.cc:2229] T ee565d95b1b5491bb2765d001fcc0cd2 P 1c6be47ad34f432987a5d2ef28f0123f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:17:42.615809 17255 raft_consensus.cc:2258] T ee565d95b1b5491bb2765d001fcc0cd2 P 1c6be47ad34f432987a5d2ef28f0123f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:17:42.616989 17255 ts_tablet_manager.cc:1905] T ee565d95b1b5491bb2765d001fcc0cd2 P 1c6be47ad34f432987a5d2ef28f0123f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:17:42.618593 17255 ts_tablet_manager.cc:1918] T ee565d95b1b5491bb2765d001fcc0cd2 P 1c6be47ad34f432987a5d2ef28f0123f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:17:42.618670 17255 log.cc:1196] T ee565d95b1b5491bb2765d001fcc0cd2 P 1c6be47ad34f432987a5d2ef28f0123f: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-2/wal/wals/ee565d95b1b5491bb2765d001fcc0cd2
I20260926 06:17:42.618896 17255 ts_tablet_manager.cc:1939] T ee565d95b1b5491bb2765d001fcc0cd2 P 1c6be47ad34f432987a5d2ef28f0123f: Deleting consensus metadata
I20260926 06:17:42.619300 16751 catalog_manager.cc:4841] TS 1c6be47ad34f432987a5d2ef28f0123f (127.2.110.195:33005): tablet ee565d95b1b5491bb2765d001fcc0cd2 (table test-table [id=301fda98571345e0a0129e7d4145dad4]) successfully deleted
I20260926 06:17:42.621490 17254 tablet_replica.cc:330] T ee565d95b1b5491bb2765d001fcc0cd2 P deaf547998bd4cce8d3970dce1eb51ad: stopping tablet replica
I20260926 06:17:42.621628 17254 raft_consensus.cc:2229] T ee565d95b1b5491bb2765d001fcc0cd2 P deaf547998bd4cce8d3970dce1eb51ad [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:17:42.621745 17254 raft_consensus.cc:2258] T ee565d95b1b5491bb2765d001fcc0cd2 P deaf547998bd4cce8d3970dce1eb51ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:17:42.622231 17254 ts_tablet_manager.cc:1905] T ee565d95b1b5491bb2765d001fcc0cd2 P deaf547998bd4cce8d3970dce1eb51ad: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 06:17:42.622957 16751 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.110.193:43911 (error 108)
I20260926 06:17:42.623476 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 16960
I20260926 06:17:42.623558 16962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:42.623429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.193:43911 (local address 127.0.0.1:47532)
0926 06:17:42.623486 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:42.623520 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.193:43911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 06:17:42.623754 16751 catalog_manager.cc:4584] TS 8778addf8daf4894b885ea833fac225c (127.2.110.193:43911): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ee565d95b1b5491bb2765d001fcc0cd2: Network error: Client connection negotiation failed: client connection to 127.2.110.193:43911: connect: Connection refused (error 111)
I20260926 06:17:42.627952 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 17096
I20260926 06:17:42.628050 16962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:42.627926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.194:46487 (local address 127.0.0.1:51082)
0926 06:17:42.627968 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:42.627995 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.194:46487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260926 06:17:42.633038 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 16734
I20260926 06:17:42.638648 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:35291
--webserver_interface=127.2.110.254
--webserver_port=43605
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:43301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.110.254:35291
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:17:42.724677 17257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:17:42.724901 17257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:17:42.724925 17257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:17:42.726171 17257 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:17:42.726215 17257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:17:42.726230 17257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:17:42.726243 17257 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:17:42.726255 17257 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:17:42.727756 17257 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:43301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.110.254:35291
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:35291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.110.254
--webserver_port=43605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:17:42.728025 17257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:17:42.728272 17257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:17:42.733637 17263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:17:42.733628 17265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:17:42.733632 17262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:17:42.733976 17257 server_base.cc:1034] running on GCE node
I20260926 06:17:42.734269 17257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:17:42.734566 17257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:17:42.735742 17257 hybrid_clock.cc:648] HybridClock initialized: now 1790403462735730 us; error 40 us; skew 500 ppm
Sep 26 06:17:42 dist-test-slave-hndd krb5kdc[16186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403462, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:17:42.738358 17257 init.cc:377] Logged in from keytab as kudu/127.2.110.254@KRBTEST.COM (short username kudu)
I20260926 06:17:42.738792 17257 webserver.cc:460] Webserver started at http://127.2.110.254:43605/ using document root <none> and password file <none>
I20260926 06:17:42.739035 17257 fs_manager.cc:359] Metadata directory not provided
I20260926 06:17:42.739106 17257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:17:42.740471 17257 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:17:42.741194 17272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:17:42.741396 17257 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:17:42.741508 17257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "a58d836c12134ed295238497e9f8f94d"
format_stamp: "Formatted at 2026-09-26 06:17:41 on dist-test-slave-hndd"
server_key: "a7204bff9c35a98b89c459ba365ae40a"
server_key_iv: "6cc563169f4155333caa39a55ad03acf"
server_key_version: "encryptionkey@0"
I20260926 06:17:42.741871 17257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:17:42.750644 17257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:17:42.752931 17257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:17:42.753096 17257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:17:42.757218 17257 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.254:35291
I20260926 06:17:42.757270 17324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.254:35291 every 8 connection(s)
I20260926 06:17:42.757688 17257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:17:42.759184 17325 sys_catalog.cc:263] Verifying existing consensus state
I20260926 06:17:42.759840 17325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d: Bootstrap starting.
I20260926 06:17:42.765587 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 17257
I20260926 06:17:42.765902 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.193:43911
--local_ip_for_outbound_sockets=127.2.110.193
--tserver_master_addrs=127.2.110.254:35291
--webserver_port=44895
--webserver_interface=127.2.110.193
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab
--principal=kudu/127.2.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:43301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 06:17:42.768262 17325 log.cc:824] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d: Log is configured to *not* fsync() on all Append() calls
I20260926 06:17:42.770582 17325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 06:17:42.770887 17325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d: Bootstrap complete.
I20260926 06:17:42.773088 17325 raft_consensus.cc:348] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58d836c12134ed295238497e9f8f94d" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 35291 } }
I20260926 06:17:42.773335 17325 raft_consensus.cc:729] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a58d836c12134ed295238497e9f8f94d, State: Initialized, Role: FOLLOWER
I20260926 06:17:42.773541 17325 consensus_queue.cc:260] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58d836c12134ed295238497e9f8f94d" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 35291 } }
I20260926 06:17:42.773623 17325 raft_consensus.cc:388] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:17:42.773679 17325 raft_consensus.cc:482] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:17:42.773730 17325 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d [term 2 FOLLOWER]: Advancing to term 3
I20260926 06:17:42.774715 17325 raft_consensus.cc:504] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58d836c12134ed295238497e9f8f94d" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 35291 } }
I20260926 06:17:42.774839 17325 leader_election.cc:302] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a58d836c12134ed295238497e9f8f94d; no voters:
I20260926 06:17:42.775053 17325 leader_election.cc:288] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 06:17:42.775092 17329 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d [term 3 FOLLOWER]: Leader election won for term 3
I20260926 06:17:42.775202 17329 raft_consensus.cc:686] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d [term 3 LEADER]: Becoming Leader. State: Replica: a58d836c12134ed295238497e9f8f94d, State: Running, Role: LEADER
I20260926 06:17:42.775298 17329 consensus_queue.cc:237] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58d836c12134ed295238497e9f8f94d" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 35291 } }
I20260926 06:17:42.775357 17325 sys_catalog.cc:564] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:17:42.775568 17330 sys_catalog.cc:455] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a58d836c12134ed295238497e9f8f94d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58d836c12134ed295238497e9f8f94d" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 35291 } } }
I20260926 06:17:42.775651 17330 sys_catalog.cc:458] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d [sys.catalog]: This master's current role is: LEADER
I20260926 06:17:42.775882 17331 sys_catalog.cc:455] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a58d836c12134ed295238497e9f8f94d. Latest consensus state: current_term: 3 leader_uuid: "a58d836c12134ed295238497e9f8f94d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58d836c12134ed295238497e9f8f94d" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 35291 } } }
I20260926 06:17:42.775992 17331 sys_catalog.cc:458] T 00000000000000000000000000000000 P a58d836c12134ed295238497e9f8f94d [sys.catalog]: This master's current role is: LEADER
I20260926 06:17:42.776140 17333 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:17:42.776964 17333 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:17:42.777069 17333 catalog_manager.cc:1259] Loaded cluster ID: c35f709572054fc4a7c26e4aa20b5718
I20260926 06:17:42.777094 17333 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 06:17:42.777323 17333 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f9426af89ad google::LogMessage::Fail() at ??:0
@ 0x7f9426afb88a google::LogMessage::SendToLog() at ??:0
@ 0x7f9426af84de google::LogMessage::Flush() at ??:0
@ 0x7f9426af9a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f942ea190f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f942ea228b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f94289c60d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f94289bb954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f942981f6db start_thread at ??:0
@ 0x7f94255f661f clone at ??:0
W20260926 06:17:42.884227 17327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:17:42.884510 17327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:17:42.884552 17327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:17:42.886420 17327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:17:42.886492 17327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:17:42.886538 17327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.193
I20260926 06:17:42.889143 17327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:43301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab
--principal=kudu/127.2.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.193:43911
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.110.193
--webserver_port=44895
--tserver_master_addrs=127.2.110.254:35291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.193
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:17:42.889470 17327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:17:42.889838 17327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:17:42.895354 17364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:17:42.895293 17361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:17:42.895287 17362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:17:42.895587 17327 server_base.cc:1034] running on GCE node
I20260926 06:17:42.895808 17327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:17:42.896075 17327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:17:42.897244 17327 hybrid_clock.cc:648] HybridClock initialized: now 1790403462897217 us; error 42 us; skew 500 ppm
Sep 26 06:17:42 dist-test-slave-hndd krb5kdc[16186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403462, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:17:42.899770 17327 init.cc:377] Logged in from keytab as kudu/127.2.110.193@KRBTEST.COM (short username kudu)
I20260926 06:17:42.900215 17327 webserver.cc:460] Webserver started at http://127.2.110.193:44895/ using document root <none> and password file <none>
I20260926 06:17:42.900473 17327 fs_manager.cc:359] Metadata directory not provided
I20260926 06:17:42.900547 17327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:17:42.902047 17327 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:17:42.902793 17371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:17:42.902973 17327 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:17:42.903075 17327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-0/wal
uuid: "8778addf8daf4894b885ea833fac225c"
format_stamp: "Formatted at 2026-09-26 06:17:41 on dist-test-slave-hndd"
server_key: "dbe7afcd73ffde686eb3744261deb453"
server_key_iv: "f2ae0f5d2705ce8bfce82f650158c135"
server_key_version: "encryptionkey@0"
I20260926 06:17:42.903445 17327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:17:42.917546 17327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:17:42.919889 17327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:17:42.920069 17327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:17:42.920316 17327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:17:42.920789 17378 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 06:17:42.922765 17327 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 06:17:42.922827 17327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 06:17:42.922885 17327 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 06:17:42.923527 17327 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 06:17:42.923588 17327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 06:17:42.923657 17378 tablet_bootstrap.cc:492] T ee565d95b1b5491bb2765d001fcc0cd2 P 8778addf8daf4894b885ea833fac225c: Bootstrap starting.
I20260926 06:17:42.931407 17327 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.193:43911
I20260926 06:17:42.931641 17485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.193:43911 every 8 connection(s)
I20260926 06:17:42.931957 17327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
I20260926 06:17:42.934746 17378 log.cc:824] T ee565d95b1b5491bb2765d001fcc0cd2 P 8778addf8daf4894b885ea833fac225c: Log is configured to *not* fsync() on all Append() calls
I20260926 06:17:42.937481 17378 tablet_bootstrap.cc:492] T ee565d95b1b5491bb2765d001fcc0cd2 P 8778addf8daf4894b885ea833fac225c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 06:17:42.937831 17378 tablet_bootstrap.cc:492] T ee565d95b1b5491bb2765d001fcc0cd2 P 8778addf8daf4894b885ea833fac225c: Bootstrap complete.
I20260926 06:17:42.939073 17378 ts_tablet_manager.cc:1397] T ee565d95b1b5491bb2765d001fcc0cd2 P 8778addf8daf4894b885ea833fac225c: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.003s
I20260926 06:17:42.940541 17378 raft_consensus.cc:348] T ee565d95b1b5491bb2765d001fcc0cd2 P 8778addf8daf4894b885ea833fac225c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c6be47ad34f432987a5d2ef28f0123f" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 33005 } } peers { permanent_uuid: "deaf547998bd4cce8d3970dce1eb51ad" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 46487 } } peers { permanent_uuid: "8778addf8daf4894b885ea833fac225c" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 43911 } }
I20260926 06:17:42.940896 17378 raft_consensus.cc:729] T ee565d95b1b5491bb2765d001fcc0cd2 P 8778addf8daf4894b885ea833fac225c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8778addf8daf4894b885ea833fac225c, State: Initialized, Role: FOLLOWER
I20260926 06:17:42.941047 17378 consensus_queue.cc:260] T ee565d95b1b5491bb2765d001fcc0cd2 P 8778addf8daf4894b885ea833fac225c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c6be47ad34f432987a5d2ef28f0123f" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 33005 } } peers { permanent_uuid: "deaf547998bd4cce8d3970dce1eb51ad" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 46487 } } peers { permanent_uuid: "8778addf8daf4894b885ea833fac225c" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 43911 } }
I20260926 06:17:42.941329 17378 ts_tablet_manager.cc:1428] T ee565d95b1b5491bb2765d001fcc0cd2 P 8778addf8daf4894b885ea833fac225c: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
I20260926 06:17:42.941777 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 17327
I20260926 06:17:42.942029 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.194:46487
--local_ip_for_outbound_sockets=127.2.110.194
--tserver_master_addrs=127.2.110.254:35291
--webserver_port=34939
--webserver_interface=127.2.110.194
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab
--principal=kudu/127.2.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:43301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:17:43.038986 17488 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:17:42.932803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.193:44615)
0926 06:17:42.932990 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:42.933035 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 06:17:42.933580 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:17:43.038764 (+105184us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:35291: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":100,"threads_started":1}
W20260926 06:17:43.039280 17486 heartbeater.cc:644] Failed to heartbeat to 127.2.110.254:35291 (0 consecutive failures): Network error: Failed to ping master at 127.2.110.254:35291: Client connection negotiation failed: client connection to 127.2.110.254:35291: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 06:17:43.039615 17488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:43.039492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.193:59293)
0926 06:17:43.039548 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:43.039587 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 06:17:43.040000 17488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:43.039913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.193:42979)
0926 06:17:43.039963 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:43.039986 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 06:17:43.040132 17486 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 06:17:43.050422 17491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:17:43.050673 17491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:17:43.050719 17491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:17:43.051963 17491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:17:43.052035 17491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:17:43.052083 17491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.194
I20260926 06:17:43.053709 17491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:43301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab
--principal=kudu/127.2.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.194:46487
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.110.194
--webserver_port=34939
--tserver_master_addrs=127.2.110.254:35291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.194
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:17:43.054003 17491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:17:43.054250 17491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:17:43.059938 17496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:17:43.059954 17499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:17:43.059983 17497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:17:43.059984 17491 server_base.cc:1034] running on GCE node
I20260926 06:17:43.060371 17491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:17:43.060577 17491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:17:43.061749 17491 hybrid_clock.cc:648] HybridClock initialized: now 1790403463061737 us; error 40 us; skew 500 ppm
Sep 26 06:17:43 dist-test-slave-hndd krb5kdc[16186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403463, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:17:43.064429 17491 init.cc:377] Logged in from keytab as kudu/127.2.110.194@KRBTEST.COM (short username kudu)
I20260926 06:17:43.064983 17491 webserver.cc:460] Webserver started at http://127.2.110.194:34939/ using document root <none> and password file <none>
I20260926 06:17:43.065238 17491 fs_manager.cc:359] Metadata directory not provided
I20260926 06:17:43.065310 17491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:17:43.066769 17491 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:17:43.067576 17506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:17:43.067762 17491 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:17:43.067863 17491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-1/wal
uuid: "deaf547998bd4cce8d3970dce1eb51ad"
format_stamp: "Formatted at 2026-09-26 06:17:41 on dist-test-slave-hndd"
server_key: "c565334f1559a7d546f2685326074869"
server_key_iv: "e029a9d7333aa4497ca7145f386b377f"
server_key_version: "encryptionkey@0"
I20260926 06:17:43.068239 17491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:17:43.087036 17491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:17:43.089511 17491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:17:43.089685 17491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:17:43.089941 17491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:17:43.090392 17513 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260926 06:17:43.091822 17513 ts_tablet_manager.cc:1852] T ee565d95b1b5491bb2765d001fcc0cd2 P deaf547998bd4cce8d3970dce1eb51ad: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260926 06:17:43.092104 17513 ts_tablet_manager.cc:1905] T ee565d95b1b5491bb2765d001fcc0cd2 P deaf547998bd4cce8d3970dce1eb51ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:17:43.094153 17513 ts_tablet_manager.cc:1918] T ee565d95b1b5491bb2765d001fcc0cd2 P deaf547998bd4cce8d3970dce1eb51ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:17:43.094231 17513 log.cc:1196] T ee565d95b1b5491bb2765d001fcc0cd2 P deaf547998bd4cce8d3970dce1eb51ad: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-1/wal/wals/ee565d95b1b5491bb2765d001fcc0cd2
I20260926 06:17:43.094513 17513 ts_tablet_manager.cc:1939] T ee565d95b1b5491bb2765d001fcc0cd2 P deaf547998bd4cce8d3970dce1eb51ad: Deleting consensus metadata
I20260926 06:17:43.094743 17491 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260926 06:17:43.094825 17491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260926 06:17:43.094877 17491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:17:43.094923 17491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:17:43.101018 17491 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.194:46487
I20260926 06:17:43.101075 17620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.194:46487 every 8 connection(s)
I20260926 06:17:43.101392 17491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
I20260926 06:17:43.102394 17623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:43.102035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.194:33159)
0926 06:17:43.102191 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:43.102260 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":71,"threads_started":1}
W20260926 06:17:43.102628 17621 heartbeater.cc:644] Failed to heartbeat to 127.2.110.254:35291 (0 consecutive failures): Network error: Failed to ping master at 127.2.110.254:35291: Client connection negotiation failed: client connection to 127.2.110.254:35291: connect: Connection refused (error 111)
I20260926 06:17:43.102917 17623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:43.102828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.194:56999)
0926 06:17:43.102876 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:43.102900 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 06:17:43.103163 17623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:43.103116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.194:41927)
0926 06:17:43.103136 (+ 20us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:43.103152 (+ 16us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260926 06:17:43.103309 17621 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 06:17:43.108613 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 17491
I20260926 06:17:43.108891 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.195:33005
--local_ip_for_outbound_sockets=127.2.110.195
--tserver_master_addrs=127.2.110.254:35291
--webserver_port=38621
--webserver_interface=127.2.110.195
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab
--principal=kudu/127.2.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:43301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:17:43.193456 17624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:17:43.193681 17624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:17:43.193706 17624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:17:43.194895 17624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:17:43.194945 17624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:17:43.194974 17624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.195
I20260926 06:17:43.196560 17624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:43301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab
--principal=kudu/127.2.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.195:33005
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.110.195
--webserver_port=38621
--tserver_master_addrs=127.2.110.254:35291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.195
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:17:43.196825 17624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:17:43.197072 17624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:17:43.203250 17630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:17:43.203384 17629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:17:43.203250 17632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:17:43.203578 17624 server_base.cc:1034] running on GCE node
I20260926 06:17:43.203794 17624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:17:43.204025 17624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:17:43.205178 17624 hybrid_clock.cc:648] HybridClock initialized: now 1790403463205164 us; error 29 us; skew 500 ppm
Sep 26 06:17:43 dist-test-slave-hndd krb5kdc[16186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403463, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:17:43.207832 17624 init.cc:377] Logged in from keytab as kudu/127.2.110.195@KRBTEST.COM (short username kudu)
I20260926 06:17:43.208298 17624 webserver.cc:460] Webserver started at http://127.2.110.195:38621/ using document root <none> and password file <none>
I20260926 06:17:43.208564 17624 fs_manager.cc:359] Metadata directory not provided
I20260926 06:17:43.208614 17624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:17:43.210119 17624 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 06:17:43.210875 17639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:17:43.211127 17624 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:17:43.211210 17624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-2/wal
uuid: "1c6be47ad34f432987a5d2ef28f0123f"
format_stamp: "Formatted at 2026-09-26 06:17:41 on dist-test-slave-hndd"
server_key: "62639a18bf85d59a91212d35060eb7a6"
server_key_iv: "08e8d15d0ea5c3388dbc3b4800c2ef5d"
server_key_version: "encryptionkey@0"
I20260926 06:17:43.211596 17624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:17:43.230697 17624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:17:43.233134 17624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:17:43.233302 17624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:17:43.233660 17624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:17:43.233991 17624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:17:43.234026 17624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:17:43.234071 17624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:17:43.234103 17624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:17:43.241122 17624 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.195:33005
I20260926 06:17:43.241165 17752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.195:33005 every 8 connection(s)
I20260926 06:17:43.241568 17624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
I20260926 06:17:43.242923 17755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:43.242532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.195:33481)
0926 06:17:43.242693 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:43.242767 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":72,"threads_started":1}
W20260926 06:17:43.243407 17753 heartbeater.cc:644] Failed to heartbeat to 127.2.110.254:35291 (0 consecutive failures): Network error: Failed to ping master at 127.2.110.254:35291: Client connection negotiation failed: client connection to 127.2.110.254:35291: connect: Connection refused (error 111)
I20260926 06:17:43.243731 17755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:43.243641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.195:52093)
0926 06:17:43.243687 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:43.243714 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 06:17:43.244071 17755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:43.243954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.195:47483)
0926 06:17:43.244020 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:43.244049 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 06:17:43.244359 17753 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 06:17:43.244583 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 17624
I20260926 06:17:44.040761 17757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:44.040536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.193:60821)
0926 06:17:44.040656 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:44.040698 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":61,"threads_started":1}
I20260926 06:17:44.103894 17758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:44.103662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.194:42713)
0926 06:17:44.103814 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:44.103850 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":62,"threads_started":1}
I20260926 06:17:44.245102 17759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:44.244857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.195:40625)
0926 06:17:44.244984 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:44.245037 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":65,"threads_started":1}
I20260926 06:17:44.320235 17760 raft_consensus.cc:482] T ee565d95b1b5491bb2765d001fcc0cd2 P 8778addf8daf4894b885ea833fac225c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:17:44.320434 17760 raft_consensus.cc:504] T ee565d95b1b5491bb2765d001fcc0cd2 P 8778addf8daf4894b885ea833fac225c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c6be47ad34f432987a5d2ef28f0123f" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 33005 } } peers { permanent_uuid: "deaf547998bd4cce8d3970dce1eb51ad" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 46487 } } peers { permanent_uuid: "8778addf8daf4894b885ea833fac225c" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 43911 } }
I20260926 06:17:44.320869 17760 leader_election.cc:288] T ee565d95b1b5491bb2765d001fcc0cd2 P 8778addf8daf4894b885ea833fac225c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c6be47ad34f432987a5d2ef28f0123f (127.2.110.195:33005), deaf547998bd4cce8d3970dce1eb51ad (127.2.110.194:46487)
Sep 26 06:17:44 dist-test-slave-hndd krb5kdc[16186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403462, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.193@KRBTEST.COM for kudu/127.2.110.195@KRBTEST.COM
Sep 26 06:17:44 dist-test-slave-hndd krb5kdc[16186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403462, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.193@KRBTEST.COM for kudu/127.2.110.194@KRBTEST.COM
I20260926 06:17:44.334200 17763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:44.321422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:39873 (local address 127.2.110.195:33005)
0926 06:17:44.321566 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 06:17:44.321577 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 06:17:44.321652 (+ 75us) server_negotiation.cc:407] Connection header received
0926 06:17:44.322287 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:17:44.322289 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:17:44.322328 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:17:44.322464 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:17:44.323208 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:44.324085 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:44.325123 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:44.325601 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:44.325744 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:44.328727 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:17:44.328733 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:17:44.328733 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:17:44.328747 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:17:44.332120 (+ 3373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:44.332712 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:44.332714 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:44.332715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:44.332834 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:44.333278 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:44.333279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:44.333279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:44.333647 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:17:44.333742 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 06:17:44.333935 (+ 193us) server_negotiation.cc:299] Negotiation successful
0926 06:17:44.334077 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":74,"threads_started":1}
I20260926 06:17:44.334183 17757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:44.321099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.194:46487 (local address 127.2.110.193:42477)
0926 06:17:44.321412 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:44.321424 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:17:44.321498 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:17:44.322493 (+ 995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:17:44.322496 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:17:44.322894 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:17:44.323081 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:44.323087 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:44.324213 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:44.324214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:44.325023 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:44.325025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:44.325715 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:44.325717 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:44.325823 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:44.326819 (+ 996us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:17:44.326827 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:17:44.328921 (+ 2094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:17:44.331882 (+ 2961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:17:44.331892 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:17:44.331895 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:17:44.332295 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:17:44.332738 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:17:44.332739 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:17:44.332739 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:17:44.332881 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:17:44.333577 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:17:44.333585 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:17:44.333737 (+ 152us) client_negotiation.cc:769] Sending connection context
0926 06:17:44.333856 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 06:17:44.334075 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":86}
I20260926 06:17:44.334214 17762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:44.321195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:42477 (local address 127.2.110.194:46487)
0926 06:17:44.321394 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 06:17:44.321414 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 06:17:44.321481 (+ 67us) server_negotiation.cc:407] Connection header received
0926 06:17:44.322240 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:17:44.322242 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:17:44.322290 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:17:44.322462 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:17:44.323204 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:44.324082 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:44.325138 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:17:44.325616 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:17:44.325748 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:44.329027 (+ 3279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:17:44.329030 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:17:44.329031 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:17:44.329040 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:17:44.331783 (+ 2743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:44.332401 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:44.332404 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:44.332405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:44.332516 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:17:44.333054 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:17:44.333056 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:17:44.333056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:17:44.333494 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:17:44.333569 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 06:17:44.333933 (+ 364us) server_negotiation.cc:299] Negotiation successful
0926 06:17:44.334091 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":74,"threads_started":1}
I20260926 06:17:44.334384 17761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:44.321104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.195:33005 (local address 127.2.110.193:39873)
0926 06:17:44.321587 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:44.321598 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:17:44.321667 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:17:44.322493 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:17:44.322496 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:17:44.322896 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:17:44.323081 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:44.323087 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:44.324200 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:44.324201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:44.325023 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:17:44.325025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:17:44.325708 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:17:44.325709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:17:44.325823 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:17:44.326704 (+ 881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:17:44.326713 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:17:44.328471 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:17:44.332308 (+ 3837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:17:44.332310 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:17:44.332310 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:17:44.332624 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:17:44.333059 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:17:44.333060 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:17:44.333060 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:17:44.333170 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:17:44.333736 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:17:44.333738 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:17:44.333778 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 06:17:44.333877 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 06:17:44.334166 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"mutex_wait_us":171,"thread_start_us":81,"threads_started":1}
W20260926 06:17:44.335253 17373 leader_election.cc:341] T ee565d95b1b5491bb2765d001fcc0cd2 P 8778addf8daf4894b885ea833fac225c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer deaf547998bd4cce8d3970dce1eb51ad (127.2.110.194:46487): Not found: Tablet not found: ee565d95b1b5491bb2765d001fcc0cd2
W20260926 06:17:44.335741 17372 leader_election.cc:341] T ee565d95b1b5491bb2765d001fcc0cd2 P 8778addf8daf4894b885ea833fac225c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1c6be47ad34f432987a5d2ef28f0123f (127.2.110.195:33005): Not found: Tablet not found: ee565d95b1b5491bb2765d001fcc0cd2
I20260926 06:17:44.335804 17372 leader_election.cc:302] T ee565d95b1b5491bb2765d001fcc0cd2 P 8778addf8daf4894b885ea833fac225c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8778addf8daf4894b885ea833fac225c; no voters: 1c6be47ad34f432987a5d2ef28f0123f, deaf547998bd4cce8d3970dce1eb51ad
I20260926 06:17:44.335925 17760 raft_consensus.cc:2726] T ee565d95b1b5491bb2765d001fcc0cd2 P 8778addf8daf4894b885ea833fac225c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 06:17:45.041617 17764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:45.041367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.193:56215)
0926 06:17:45.041512 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:45.041559 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":65,"threads_started":1}
I20260926 06:17:45.104934 17765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:45.104656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.194:47489)
0926 06:17:45.104824 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:45.104890 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":73,"threads_started":1}
I20260926 06:17:45.245846 17766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:45.245616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.195:36617)
0926 06:17:45.245742 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:45.245795 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":58,"threads_started":1}
I20260926 06:17:46.042507 17767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:46.042278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.193:36097)
0926 06:17:46.042417 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:46.042460 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":64,"threads_started":1}
I20260926 06:17:46.105782 17768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:46.105509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.194:41357)
0926 06:17:46.105682 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:46.105735 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":110,"threads_started":1}
I20260926 06:17:46.246670 17769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:46.246416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.195:45257)
0926 06:17:46.246577 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:46.246620 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":66,"threads_started":1}
I20260926 06:17:46.577464 17770 raft_consensus.cc:482] T ee565d95b1b5491bb2765d001fcc0cd2 P 8778addf8daf4894b885ea833fac225c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:17:46.577575 17770 raft_consensus.cc:504] T ee565d95b1b5491bb2765d001fcc0cd2 P 8778addf8daf4894b885ea833fac225c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c6be47ad34f432987a5d2ef28f0123f" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 33005 } } peers { permanent_uuid: "deaf547998bd4cce8d3970dce1eb51ad" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 46487 } } peers { permanent_uuid: "8778addf8daf4894b885ea833fac225c" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 43911 } }
I20260926 06:17:46.577775 17770 leader_election.cc:288] T ee565d95b1b5491bb2765d001fcc0cd2 P 8778addf8daf4894b885ea833fac225c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c6be47ad34f432987a5d2ef28f0123f (127.2.110.195:33005), deaf547998bd4cce8d3970dce1eb51ad (127.2.110.194:46487)
W20260926 06:17:46.578231 17372 leader_election.cc:341] T ee565d95b1b5491bb2765d001fcc0cd2 P 8778addf8daf4894b885ea833fac225c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1c6be47ad34f432987a5d2ef28f0123f (127.2.110.195:33005): Not found: Tablet not found: ee565d95b1b5491bb2765d001fcc0cd2
W20260926 06:17:46.578310 17373 leader_election.cc:341] T ee565d95b1b5491bb2765d001fcc0cd2 P 8778addf8daf4894b885ea833fac225c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer deaf547998bd4cce8d3970dce1eb51ad (127.2.110.194:46487): Not found: Tablet not found: ee565d95b1b5491bb2765d001fcc0cd2
I20260926 06:17:46.578387 17373 leader_election.cc:302] T ee565d95b1b5491bb2765d001fcc0cd2 P 8778addf8daf4894b885ea833fac225c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8778addf8daf4894b885ea833fac225c; no voters: 1c6be47ad34f432987a5d2ef28f0123f, deaf547998bd4cce8d3970dce1eb51ad
I20260926 06:17:46.578490 17770 raft_consensus.cc:2726] T ee565d95b1b5491bb2765d001fcc0cd2 P 8778addf8daf4894b885ea833fac225c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 06:17:47.043334 17771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:47.043050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.193:40615)
0926 06:17:47.043225 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:47.043286 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":81,"threads_started":1}
I20260926 06:17:47.106703 17772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:47.106417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.194:59545)
0926 06:17:47.106592 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:47.106643 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":90,"threads_started":1}
I20260926 06:17:47.247496 17773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:47.247239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.195:59627)
0926 06:17:47.247408 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:47.247450 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":67,"threads_started":1}
I20260926 06:17:48.044198 17774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:48.043942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.193:47963)
0926 06:17:48.044088 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:48.044134 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":69,"threads_started":1}
I20260926 06:17:48.107517 17775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:48.107313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.194:59957)
0926 06:17:48.107436 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:48.107471 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":63,"threads_started":1}
I20260926 06:17:48.248229 17776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:48.247968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.195:39943)
0926 06:17:48.248138 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:48.248177 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":74,"threads_started":1}
I20260926 06:17:48.432789 17777 raft_consensus.cc:482] T ee565d95b1b5491bb2765d001fcc0cd2 P 8778addf8daf4894b885ea833fac225c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:17:48.432919 17777 raft_consensus.cc:504] T ee565d95b1b5491bb2765d001fcc0cd2 P 8778addf8daf4894b885ea833fac225c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c6be47ad34f432987a5d2ef28f0123f" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 33005 } } peers { permanent_uuid: "deaf547998bd4cce8d3970dce1eb51ad" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 46487 } } peers { permanent_uuid: "8778addf8daf4894b885ea833fac225c" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 43911 } }
I20260926 06:17:48.433126 17777 leader_election.cc:288] T ee565d95b1b5491bb2765d001fcc0cd2 P 8778addf8daf4894b885ea833fac225c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c6be47ad34f432987a5d2ef28f0123f (127.2.110.195:33005), deaf547998bd4cce8d3970dce1eb51ad (127.2.110.194:46487)
W20260926 06:17:48.433599 17373 leader_election.cc:341] T ee565d95b1b5491bb2765d001fcc0cd2 P 8778addf8daf4894b885ea833fac225c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer deaf547998bd4cce8d3970dce1eb51ad (127.2.110.194:46487): Not found: Tablet not found: ee565d95b1b5491bb2765d001fcc0cd2
W20260926 06:17:48.433676 17372 leader_election.cc:341] T ee565d95b1b5491bb2765d001fcc0cd2 P 8778addf8daf4894b885ea833fac225c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1c6be47ad34f432987a5d2ef28f0123f (127.2.110.195:33005): Not found: Tablet not found: ee565d95b1b5491bb2765d001fcc0cd2
I20260926 06:17:48.433777 17372 leader_election.cc:302] T ee565d95b1b5491bb2765d001fcc0cd2 P 8778addf8daf4894b885ea833fac225c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8778addf8daf4894b885ea833fac225c; no voters: 1c6be47ad34f432987a5d2ef28f0123f, deaf547998bd4cce8d3970dce1eb51ad
I20260926 06:17:48.433845 17777 raft_consensus.cc:2726] T ee565d95b1b5491bb2765d001fcc0cd2 P 8778addf8daf4894b885ea833fac225c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 06:17:49.045548 17778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:49.044764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.193:45901)
0926 06:17:49.045215 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:49.045324 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":80,"threads_started":1}
I20260926 06:17:49.108487 17779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:49.108196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.194:34163)
0926 06:17:49.108378 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:49.108440 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":95,"threads_started":1}
I20260926 06:17:49.248986 17780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:49.248745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.195:56281)
0926 06:17:49.248900 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:49.248938 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":79,"threads_started":1}
I20260926 06:17:50.046438 17781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:50.046159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.193:50325)
0926 06:17:50.046338 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:50.046387 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":67,"threads_started":1}
I20260926 06:17:50.109504 17782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:50.109176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.194:38869)
0926 06:17:50.109408 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:50.109452 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":73,"threads_started":1}
I20260926 06:17:50.249939 17783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:50.249619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.195:59375)
0926 06:17:50.249837 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:50.249881 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":83,"threads_started":1}
I20260926 06:17:51.047319 17784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:51.047068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.193:35755)
0926 06:17:51.047212 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:51.047271 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":71,"threads_started":1}
I20260926 06:17:51.110406 17785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:51.110154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.194:37619)
0926 06:17:51.110311 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:51.110359 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":68,"threads_started":1}
I20260926 06:17:51.250687 17786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:51.250446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.195:45881)
0926 06:17:51.250592 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:51.250638 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":80,"threads_started":1}
I20260926 06:17:51.327216 17787 raft_consensus.cc:482] T ee565d95b1b5491bb2765d001fcc0cd2 P 8778addf8daf4894b885ea833fac225c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:17:51.327385 17787 raft_consensus.cc:504] T ee565d95b1b5491bb2765d001fcc0cd2 P 8778addf8daf4894b885ea833fac225c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c6be47ad34f432987a5d2ef28f0123f" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 33005 } } peers { permanent_uuid: "deaf547998bd4cce8d3970dce1eb51ad" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 46487 } } peers { permanent_uuid: "8778addf8daf4894b885ea833fac225c" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 43911 } }
I20260926 06:17:51.327711 17787 leader_election.cc:288] T ee565d95b1b5491bb2765d001fcc0cd2 P 8778addf8daf4894b885ea833fac225c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c6be47ad34f432987a5d2ef28f0123f (127.2.110.195:33005), deaf547998bd4cce8d3970dce1eb51ad (127.2.110.194:46487)
W20260926 06:17:51.328197 17372 leader_election.cc:341] T ee565d95b1b5491bb2765d001fcc0cd2 P 8778addf8daf4894b885ea833fac225c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1c6be47ad34f432987a5d2ef28f0123f (127.2.110.195:33005): Not found: Tablet not found: ee565d95b1b5491bb2765d001fcc0cd2
W20260926 06:17:51.328336 17373 leader_election.cc:341] T ee565d95b1b5491bb2765d001fcc0cd2 P 8778addf8daf4894b885ea833fac225c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer deaf547998bd4cce8d3970dce1eb51ad (127.2.110.194:46487): Not found: Tablet not found: ee565d95b1b5491bb2765d001fcc0cd2
I20260926 06:17:51.328421 17373 leader_election.cc:302] T ee565d95b1b5491bb2765d001fcc0cd2 P 8778addf8daf4894b885ea833fac225c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8778addf8daf4894b885ea833fac225c; no voters: 1c6be47ad34f432987a5d2ef28f0123f, deaf547998bd4cce8d3970dce1eb51ad
I20260926 06:17:51.328516 17787 raft_consensus.cc:2726] T ee565d95b1b5491bb2765d001fcc0cd2 P 8778addf8daf4894b885ea833fac225c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 06:17:52.048513 17788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:52.048175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.193:37773)
0926 06:17:52.048416 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:52.048467 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":119,"threads_started":1}
I20260926 06:17:52.111284 17789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:52.111033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.194:57349)
0926 06:17:52.111186 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:52.111229 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":61,"threads_started":1}
I20260926 06:17:52.251724 17790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:52.251401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.195:46619)
0926 06:17:52.251588 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:52.251655 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":91,"threads_started":1}
I20260926 06:17:53.049521 17791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:53.049252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.193:41111)
0926 06:17:53.049420 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:53.049470 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"spinlock_wait_cycles":896,"thread_start_us":143,"threads_started":1}
I20260926 06:17:53.112141 17792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:53.111890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.194:33481)
0926 06:17:53.112047 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:53.112093 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":64,"threads_started":1}
I20260926 06:17:53.252799 17793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:53.252521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.195:53495)
0926 06:17:53.252696 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:53.252748 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":64,"threads_started":1}
I20260926 06:17:54.050390 17794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:54.050103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.193:49731)
0926 06:17:54.050285 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:54.050334 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":71,"threads_started":1}
I20260926 06:17:54.113792 17795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:54.113446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.194:49199)
0926 06:17:54.113640 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:54.113699 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":69,"threads_started":1}
I20260926 06:17:54.253763 17796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:54.253520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.195:40759)
0926 06:17:54.253668 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:54.253713 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":99,"threads_started":1}
I20260926 06:17:54.796094 17797 raft_consensus.cc:482] T ee565d95b1b5491bb2765d001fcc0cd2 P 8778addf8daf4894b885ea833fac225c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:17:54.796222 17797 raft_consensus.cc:504] T ee565d95b1b5491bb2765d001fcc0cd2 P 8778addf8daf4894b885ea833fac225c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c6be47ad34f432987a5d2ef28f0123f" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 33005 } } peers { permanent_uuid: "deaf547998bd4cce8d3970dce1eb51ad" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 46487 } } peers { permanent_uuid: "8778addf8daf4894b885ea833fac225c" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 43911 } }
I20260926 06:17:54.796468 17797 leader_election.cc:288] T ee565d95b1b5491bb2765d001fcc0cd2 P 8778addf8daf4894b885ea833fac225c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c6be47ad34f432987a5d2ef28f0123f (127.2.110.195:33005), deaf547998bd4cce8d3970dce1eb51ad (127.2.110.194:46487)
W20260926 06:17:54.796857 17372 leader_election.cc:341] T ee565d95b1b5491bb2765d001fcc0cd2 P 8778addf8daf4894b885ea833fac225c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1c6be47ad34f432987a5d2ef28f0123f (127.2.110.195:33005): Not found: Tablet not found: ee565d95b1b5491bb2765d001fcc0cd2
W20260926 06:17:54.797029 17373 leader_election.cc:341] T ee565d95b1b5491bb2765d001fcc0cd2 P 8778addf8daf4894b885ea833fac225c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer deaf547998bd4cce8d3970dce1eb51ad (127.2.110.194:46487): Not found: Tablet not found: ee565d95b1b5491bb2765d001fcc0cd2
I20260926 06:17:54.797102 17373 leader_election.cc:302] T ee565d95b1b5491bb2765d001fcc0cd2 P 8778addf8daf4894b885ea833fac225c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8778addf8daf4894b885ea833fac225c; no voters: 1c6be47ad34f432987a5d2ef28f0123f, deaf547998bd4cce8d3970dce1eb51ad
I20260926 06:17:54.797266 17797 raft_consensus.cc:2726] T ee565d95b1b5491bb2765d001fcc0cd2 P 8778addf8daf4894b885ea833fac225c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 06:17:55.051324 17798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:55.051033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.193:59619)
0926 06:17:55.051208 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:55.051256 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":93,"threads_started":1}
I20260926 06:17:55.114826 17799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:55.114537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.194:59221)
0926 06:17:55.114686 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:55.114766 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":65,"threads_started":1}
I20260926 06:17:55.254660 17800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:55.254317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.195:56373)
0926 06:17:55.254539 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:55.254593 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":100,"threads_started":1}
I20260926 06:17:56.052271 17801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:56.051952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.193:50567)
0926 06:17:56.052153 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:56.052221 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":69,"threads_started":1}
I20260926 06:17:56.115774 17802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:56.115466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.194:59193)
0926 06:17:56.115679 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:56.115728 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":110,"threads_started":1}
I20260926 06:17:56.255528 17803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:56.255281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.195:56663)
0926 06:17:56.255435 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:56.255478 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":67,"threads_started":1}
I20260926 06:17:57.053073 17804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:57.052822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.193:39363)
0926 06:17:57.052982 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:57.053027 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":80,"threads_started":1}
I20260926 06:17:57.116614 17805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:57.116380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.194:54185)
0926 06:17:57.116521 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:57.116568 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":66,"threads_started":1}
I20260926 06:17:57.256532 17806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:57.256180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.195:42685)
0926 06:17:57.256401 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:57.256467 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":113,"threads_started":1}
I20260926 06:17:58.053974 17807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:58.053706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.193:58883)
0926 06:17:58.053868 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:58.053928 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":68,"threads_started":1}
I20260926 06:17:58.117534 17808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:58.117233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.194:56229)
0926 06:17:58.117437 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:58.117484 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":65,"threads_started":1}
I20260926 06:17:58.256660 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 17327
I20260926 06:17:58.258059 17809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:17:58.257699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:35291 (local address 127.2.110.195:41725)
0926 06:17:58.257928 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0926 06:17:58.257968 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":80,"threads_started":1}
I20260926 06:17:58.262907 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 17491
I20260926 06:17:58.267843 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 17624
2026-09-26T06:17:58Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19476 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:17:59 dist-test-slave-hndd krb5kdc[17812](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:17:59 dist-test-slave-hndd krb5kdc[17812](info): set up 2 sockets
Sep 26 06:17:59 dist-test-slave-hndd krb5kdc[17812](info): commencing operation
krb5kdc: starting...
W20260926 06:18:00.301870 2491 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Sep 26 06:18:00 dist-test-slave-hndd krb5kdc[17812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:18:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:18:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
I20260926 06:18:00.432874 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:42699
--webserver_interface=127.2.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:36009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.110.254:42699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:18:00.515960 17825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:18:00.516187 17825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:18:00.516212 17825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:18:00.517511 17825 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 06:18:00.517560 17825 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 06:18:00.517575 17825 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 06:18:00.517586 17825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:18:00.517601 17825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:18:00.517613 17825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:18:00.517668 17825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:18:00.517693 17825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:18:00.519249 17825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:36009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.110.254:42699
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:42699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:18:00.519544 17825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:18:00.519811 17825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:18:00.525398 17833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:18:00.525404 17831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:18:00.525437 17825 server_base.cc:1034] running on GCE node
W20260926 06:18:00.525449 17830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:18:00.525902 17825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:18:00.526194 17825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:18:00.527359 17825 hybrid_clock.cc:648] HybridClock initialized: now 1790403480527346 us; error 32 us; skew 500 ppm
Sep 26 06:18:00 dist-test-slave-hndd krb5kdc[17812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403480, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:18:00.530066 17825 init.cc:377] Logged in from keytab as kudu/127.2.110.254@KRBTEST.COM (short username kudu)
I20260926 06:18:00.530558 17825 webserver.cc:460] Webserver started at http://127.2.110.254:40105/ using document root <none> and password file <none>
I20260926 06:18:00.530798 17825 fs_manager.cc:359] Metadata directory not provided
I20260926 06:18:00.530844 17825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:18:00.531034 17825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:18:00.532157 17825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/master-0/data/instance:
uuid: "c8235472aa12413abcb030e25ed11bfc"
format_stamp: "Formatted at 2026-09-26 06:18:00 on dist-test-slave-hndd"
server_key: "272bd0b7ba819a640db60dc1f473c9f6"
server_key_iv: "e3ddca0877505d1ddd96d5ad5f7f11c4"
server_key_version: "encryptionkey@0"
I20260926 06:18:00.532534 17825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/master-0/wal/instance:
uuid: "c8235472aa12413abcb030e25ed11bfc"
format_stamp: "Formatted at 2026-09-26 06:18:00 on dist-test-slave-hndd"
server_key: "272bd0b7ba819a640db60dc1f473c9f6"
server_key_iv: "e3ddca0877505d1ddd96d5ad5f7f11c4"
server_key_version: "encryptionkey@0"
I20260926 06:18:00.534121 17825 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:18:00.535009 17840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:18:00.535257 17825 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:18:00.535372 17825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "c8235472aa12413abcb030e25ed11bfc"
format_stamp: "Formatted at 2026-09-26 06:18:00 on dist-test-slave-hndd"
server_key: "272bd0b7ba819a640db60dc1f473c9f6"
server_key_iv: "e3ddca0877505d1ddd96d5ad5f7f11c4"
server_key_version: "encryptionkey@0"
I20260926 06:18:00.535454 17825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:18:00.557816 17825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:18:00.560209 17825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:18:00.560377 17825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:18:00.564776 17825 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.254:42699
I20260926 06:18:00.564838 17892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.254:42699 every 8 connection(s)
I20260926 06:18:00.565183 17825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:18:00.566087 17893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:18:00.568444 17893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc: Bootstrap starting.
I20260926 06:18:00.568475 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 17825
I20260926 06:18:00.568588 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/master-0/wal/instance
I20260926 06:18:00.568785 2491 external_mini_cluster.cc:1428] Setting key 0d01fa9d90abb04e279c27ebde59e3dc
I20260926 06:18:00.569129 17893 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc: Neither blocks nor log segments found. Creating new log.
I20260926 06:18:00.569497 17893 log.cc:824] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc: Log is configured to *not* fsync() on all Append() calls
I20260926 06:18:00.570230 17893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc: No bootstrap required, opened a new log
I20260926 06:18:00.572103 17893 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8235472aa12413abcb030e25ed11bfc" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 42699 } }
I20260926 06:18:00.572271 17893 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:18:00.572320 17893 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8235472aa12413abcb030e25ed11bfc, State: Initialized, Role: FOLLOWER
I20260926 06:18:00.572456 17893 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8235472aa12413abcb030e25ed11bfc" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 42699 } }
I20260926 06:18:00.572553 17893 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:18:00.572597 17893 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:18:00.572669 17893 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:18:00.573455 17893 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8235472aa12413abcb030e25ed11bfc" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 42699 } }
I20260926 06:18:00.573571 17893 leader_election.cc:302] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8235472aa12413abcb030e25ed11bfc; no voters:
I20260926 06:18:00.573755 17893 leader_election.cc:288] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:18:00.573810 17898 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:18:00.574059 17893 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:18:00.575034 17898 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc [term 1 LEADER]: Becoming Leader. State: Replica: c8235472aa12413abcb030e25ed11bfc, State: Running, Role: LEADER
I20260926 06:18:00.575165 17898 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8235472aa12413abcb030e25ed11bfc" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 42699 } }
Sep 26 06:18:00 dist-test-slave-hndd krb5kdc[17812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:18:00.575574 17898 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8235472aa12413abcb030e25ed11bfc. Latest consensus state: current_term: 1 leader_uuid: "c8235472aa12413abcb030e25ed11bfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8235472aa12413abcb030e25ed11bfc" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 42699 } } }
I20260926 06:18:00.575646 17898 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc [sys.catalog]: This master's current role is: LEADER
I20260926 06:18:00.575801 17899 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8235472aa12413abcb030e25ed11bfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8235472aa12413abcb030e25ed11bfc" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 42699 } } }
I20260926 06:18:00.575838 17913 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:18:00.575887 17899 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc [sys.catalog]: This master's current role is: LEADER
I20260926 06:18:00.576380 17913 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:18:00.578439 17913 catalog_manager.cc:1347] Generated new cluster ID: 8987a235b02f488fa2d72089fdb46881
I20260926 06:18:00.578493 17913 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:18:00.580278 17896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:00.569412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42754 (local address 127.2.110.254:42699)
0926 06:18:00.569569 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 06:18:00.569579 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 06:18:00.569609 (+ 30us) server_negotiation.cc:407] Connection header received
0926 06:18:00.570071 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:18:00.570078 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:18:00.570130 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:18:00.570282 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:18:00.570911 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:00.571816 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:00.572674 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:00.573289 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:00.573436 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:00.576161 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:18:00.576166 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:18:00.576167 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:18:00.576178 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:18:00.578031 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:18:00.578515 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:18:00.578519 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:18:00.578520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:18:00.578637 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:18:00.578874 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:18:00.578875 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:18:00.578875 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:18:00.579205 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:18:00.579371 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 06:18:00.579891 (+ 520us) server_negotiation.cc:299] Negotiation successful
0926 06:18:00.580058 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":74,"threads_started":1}
I20260926 06:18:00.589395 17913 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:18:00.590214 17913 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:18:00.598328 17913 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc: Generated new TSK 0
I20260926 06:18:00.598667 17913 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab.
Entry for principal kudu/127.2.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab.
I20260926 06:18:00.628736 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.193:0
--local_ip_for_outbound_sockets=127.2.110.193
--webserver_interface=127.2.110.193
--webserver_port=0
--tserver_master_addrs=127.2.110.254:42699
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab
--principal=kudu/127.2.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:36009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:18:00.719022 17919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:18:00.719267 17919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:18:00.719306 17919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:18:00.720479 17919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:18:00.720548 17919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:18:00.720595 17919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.193
I20260926 06:18:00.722222 17919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:36009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab
--principal=kudu/127.2.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.110.193
--webserver_port=0
--tserver_master_addrs=127.2.110.254:42699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.193
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:18:00.722494 17919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:18:00.722738 17919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:18:00.728299 17924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:18:00.728314 17925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:18:00.728314 17927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:18:00.728468 17919 server_base.cc:1034] running on GCE node
I20260926 06:18:00.728775 17919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:18:00.729023 17919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:18:00.730170 17919 hybrid_clock.cc:648] HybridClock initialized: now 1790403480730154 us; error 34 us; skew 500 ppm
Sep 26 06:18:00 dist-test-slave-hndd krb5kdc[17812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403480, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:18:00.732741 17919 init.cc:377] Logged in from keytab as kudu/127.2.110.193@KRBTEST.COM (short username kudu)
I20260926 06:18:00.733243 17919 webserver.cc:460] Webserver started at http://127.2.110.193:42771/ using document root <none> and password file <none>
I20260926 06:18:00.733541 17919 fs_manager.cc:359] Metadata directory not provided
I20260926 06:18:00.733613 17919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:18:00.733762 17919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:18:00.734845 17919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-0/data/instance:
uuid: "e559cfe3ebf84f7bb26952adfa617feb"
format_stamp: "Formatted at 2026-09-26 06:18:00 on dist-test-slave-hndd"
server_key: "eeea97a697644d4849d390d10ada85c6"
server_key_iv: "463f9f6720b364bfe68e018124e93a7f"
server_key_version: "encryptionkey@0"
I20260926 06:18:00.735209 17919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-0/wal/instance:
uuid: "e559cfe3ebf84f7bb26952adfa617feb"
format_stamp: "Formatted at 2026-09-26 06:18:00 on dist-test-slave-hndd"
server_key: "eeea97a697644d4849d390d10ada85c6"
server_key_iv: "463f9f6720b364bfe68e018124e93a7f"
server_key_version: "encryptionkey@0"
I20260926 06:18:00.736596 17919 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:18:00.737429 17934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:18:00.737603 17919 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:18:00.737713 17919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-0/wal
uuid: "e559cfe3ebf84f7bb26952adfa617feb"
format_stamp: "Formatted at 2026-09-26 06:18:00 on dist-test-slave-hndd"
server_key: "eeea97a697644d4849d390d10ada85c6"
server_key_iv: "463f9f6720b364bfe68e018124e93a7f"
server_key_version: "encryptionkey@0"
I20260926 06:18:00.737802 17919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:18:00.747673 17919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:18:00.750061 17919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:18:00.750237 17919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:18:00.750512 17919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:18:00.750840 17919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:18:00.750892 17919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:18:00.750936 17919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:18:00.750962 17919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:18:00.757323 17919 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.193:41177
I20260926 06:18:00.757385 18047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.193:41177 every 8 connection(s)
I20260926 06:18:00.757807 17919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
Sep 26 06:18:00 dist-test-slave-hndd krb5kdc[17812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403480, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.193@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:18:00.765184 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 17919
I20260926 06:18:00.765303 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-0/wal/instance
I20260926 06:18:00.765548 2491 external_mini_cluster.cc:1428] Setting key c4c0bd8cbd4e676263f9bafb20f0afec
I20260926 06:18:00.768680 17896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:00.758566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:42415 (local address 127.2.110.254:42699)
0926 06:18:00.758643 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 06:18:00.758645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:18:00.758871 (+ 226us) server_negotiation.cc:407] Connection header received
0926 06:18:00.759375 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:18:00.759376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:18:00.759387 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:18:00.759449 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:18:00.760447 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:00.760983 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:00.762147 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:00.762524 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:00.762597 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:00.765312 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:18:00.765314 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:18:00.765315 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:18:00.765329 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:18:00.766992 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:18:00.767521 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:18:00.767522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:18:00.767523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:18:00.767601 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:18:00.767916 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:18:00.767918 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:18:00.767918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:18:00.768090 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:18:00.768164 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 06:18:00.768437 (+ 273us) server_negotiation.cc:299] Negotiation successful
0926 06:18:00.768552 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 06:18:00.768698 18050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:00.758560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.193:42415)
0926 06:18:00.758743 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:00.758782 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 06:18:00.759261 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:18:00.759525 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:18:00.759529 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:18:00.760120 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:18:00.760322 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:18:00.760326 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:18:00.761091 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:18:00.761092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:18:00.761988 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:18:00.761991 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:18:00.762630 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:18:00.762631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:18:00.762751 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:00.763543 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:18:00.763553 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:18:00.765182 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:18:00.767085 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:18:00.767093 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:18:00.767096 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:18:00.767440 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:18:00.767680 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:18:00.767681 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:18:00.767681 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:18:00.767842 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:18:00.768166 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:18:00.768172 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:18:00.768301 (+ 129us) client_negotiation.cc:769] Sending connection context
0926 06:18:00.768385 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 06:18:00.768522 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":104,"threads_started":1}
I20260926 06:18:00.769253 18048 heartbeater.cc:347] Connected to a master server at 127.2.110.254:42699
I20260926 06:18:00.769380 18048 heartbeater.cc:464] Registering TS with master...
I20260926 06:18:00.769586 18048 heartbeater.cc:511] Master 127.2.110.254:42699 requested a full tablet report, sending...
I20260926 06:18:00.770238 17857 ts_manager.cc:194] Registered new tserver with Master: e559cfe3ebf84f7bb26952adfa617feb (127.2.110.193:41177)
I20260926 06:18:00.771355 17857 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.193@KRBTEST.COM'} at 127.2.110.193:42415
WARNING: no policy specified for kudu/127.2.110.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab.
Entry for principal kudu/127.2.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab.
I20260926 06:18:00.792855 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.194:0
--local_ip_for_outbound_sockets=127.2.110.194
--webserver_interface=127.2.110.194
--webserver_port=0
--tserver_master_addrs=127.2.110.254:42699
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab
--principal=kudu/127.2.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:36009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:18:00.873908 18053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:18:00.874190 18053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:18:00.874238 18053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:18:00.875558 18053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:18:00.875625 18053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:18:00.875676 18053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.194
I20260926 06:18:00.877278 18053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:36009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab
--principal=kudu/127.2.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.110.194
--webserver_port=0
--tserver_master_addrs=127.2.110.254:42699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.194
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:18:00.877585 18053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:18:00.877826 18053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:18:00.882886 18061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:18:00.882853 18058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:18:00.883082 18053 server_base.cc:1034] running on GCE node
W20260926 06:18:00.883046 18059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:18:00.883380 18053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:18:00.883783 18053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:18:00.884979 18053 hybrid_clock.cc:648] HybridClock initialized: now 1790403480884965 us; error 48 us; skew 500 ppm
Sep 26 06:18:00 dist-test-slave-hndd krb5kdc[17812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403480, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:18:00.887749 18053 init.cc:377] Logged in from keytab as kudu/127.2.110.194@KRBTEST.COM (short username kudu)
I20260926 06:18:00.888196 18053 webserver.cc:460] Webserver started at http://127.2.110.194:46171/ using document root <none> and password file <none>
I20260926 06:18:00.888389 18053 fs_manager.cc:359] Metadata directory not provided
I20260926 06:18:00.888432 18053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:18:00.888540 18053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:18:00.889659 18053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-1/data/instance:
uuid: "66143d40989641d3a0da9f5e895b22b4"
format_stamp: "Formatted at 2026-09-26 06:18:00 on dist-test-slave-hndd"
server_key: "ba5867f04f8dca91116f12cbf7de80f8"
server_key_iv: "bbcc74d120502733fe96a7a1eff028d3"
server_key_version: "encryptionkey@0"
I20260926 06:18:00.890097 18053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-1/wal/instance:
uuid: "66143d40989641d3a0da9f5e895b22b4"
format_stamp: "Formatted at 2026-09-26 06:18:00 on dist-test-slave-hndd"
server_key: "ba5867f04f8dca91116f12cbf7de80f8"
server_key_iv: "bbcc74d120502733fe96a7a1eff028d3"
server_key_version: "encryptionkey@0"
I20260926 06:18:00.891505 18053 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:18:00.892673 18068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:18:00.892899 18053 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:18:00.893002 18053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-1/wal
uuid: "66143d40989641d3a0da9f5e895b22b4"
format_stamp: "Formatted at 2026-09-26 06:18:00 on dist-test-slave-hndd"
server_key: "ba5867f04f8dca91116f12cbf7de80f8"
server_key_iv: "bbcc74d120502733fe96a7a1eff028d3"
server_key_version: "encryptionkey@0"
I20260926 06:18:00.893061 18053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:18:00.906181 18053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:18:00.908672 18053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:18:00.908790 18053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:18:00.909098 18053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:18:00.909462 18053 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:18:00.909502 18053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:18:00.909524 18053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:18:00.909538 18053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:18:00.915242 18053 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.194:41841
I20260926 06:18:00.915287 18181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.194:41841 every 8 connection(s)
I20260926 06:18:00.915613 18053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
I20260926 06:18:00.918218 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 18053
I20260926 06:18:00.918359 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-1/wal/instance
I20260926 06:18:00.918568 2491 external_mini_cluster.cc:1428] Setting key 90724dda65a7e0bb3b4538e1ddf4aad2
Sep 26 06:18:00 dist-test-slave-hndd krb5kdc[17812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403480, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.194@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:18:00.925081 17896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:00.916378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:53333 (local address 127.2.110.254:42699)
0926 06:18:00.916423 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 06:18:00.916425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:18:00.916675 (+ 250us) server_negotiation.cc:407] Connection header received
0926 06:18:00.917203 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:18:00.917204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:18:00.917215 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:18:00.917260 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:18:00.918283 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:00.918876 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:00.919866 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:00.920065 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:00.920158 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:00.922272 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:18:00.922274 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:18:00.922274 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:18:00.922283 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:18:00.923691 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:18:00.924117 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:18:00.924118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:18:00.924119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:18:00.924174 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:18:00.924427 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:18:00.924427 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:18:00.924428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:18:00.924544 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:18:00.924611 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 06:18:00.924901 (+ 290us) server_negotiation.cc:299] Negotiation successful
0926 06:18:00.924985 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 06:18:00.925272 18184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:00.916349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.194:53333)
0926 06:18:00.916547 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:00.916583 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 06:18:00.917095 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:18:00.917324 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:18:00.917376 (+ 52us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:18:00.917977 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:18:00.918180 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:18:00.918184 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:18:00.918968 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:18:00.918969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:18:00.919777 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:18:00.919779 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:18:00.920166 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:18:00.920167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:18:00.920291 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:00.920816 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:18:00.920825 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:18:00.922176 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:18:00.923773 (+ 1597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:18:00.923781 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:18:00.923783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:18:00.924015 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:18:00.924249 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:18:00.924250 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:18:00.924250 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:18:00.924357 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:18:00.924621 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:18:00.924628 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:18:00.924765 (+ 137us) client_negotiation.cc:769] Sending connection context
0926 06:18:00.924870 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 06:18:00.925037 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":109,"threads_started":1}
I20260926 06:18:00.926393 18182 heartbeater.cc:347] Connected to a master server at 127.2.110.254:42699
I20260926 06:18:00.926479 18182 heartbeater.cc:464] Registering TS with master...
I20260926 06:18:00.926687 18182 heartbeater.cc:511] Master 127.2.110.254:42699 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.110.195@KRBTEST.COM; defaulting to no policy
I20260926 06:18:00.927381 17857 ts_manager.cc:194] Registered new tserver with Master: 66143d40989641d3a0da9f5e895b22b4 (127.2.110.194:41841)
I20260926 06:18:00.927886 17857 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.194@KRBTEST.COM'} at 127.2.110.194:53333
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab.
Entry for principal kudu/127.2.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab.
I20260926 06:18:00.949739 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.195:0
--local_ip_for_outbound_sockets=127.2.110.195
--webserver_interface=127.2.110.195
--webserver_port=0
--tserver_master_addrs=127.2.110.254:42699
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab
--principal=kudu/127.2.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:36009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:18:01.036801 18187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:18:01.037070 18187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:18:01.037119 18187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:18:01.038897 18187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:18:01.038991 18187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:18:01.039060 18187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.195
I20260926 06:18:01.041397 18187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:36009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab
--principal=kudu/127.2.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.110.195
--webserver_port=0
--tserver_master_addrs=127.2.110.254:42699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.195
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:18:01.041883 18187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:18:01.042284 18187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:18:01.048029 18192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:18:01.048223 18187 server_base.cc:1034] running on GCE node
W20260926 06:18:01.048031 18193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:18:01.048072 18195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:18:01.048573 18187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:18:01.048870 18187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:18:01.050053 18187 hybrid_clock.cc:648] HybridClock initialized: now 1790403481050022 us; error 47 us; skew 500 ppm
Sep 26 06:18:01 dist-test-slave-hndd krb5kdc[17812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403481, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:18:01.052707 18187 init.cc:377] Logged in from keytab as kudu/127.2.110.195@KRBTEST.COM (short username kudu)
I20260926 06:18:01.053191 18187 webserver.cc:460] Webserver started at http://127.2.110.195:40435/ using document root <none> and password file <none>
I20260926 06:18:01.053483 18187 fs_manager.cc:359] Metadata directory not provided
I20260926 06:18:01.053534 18187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:18:01.053664 18187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:18:01.054746 18187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-2/data/instance:
uuid: "9ef9321e79e349e4af20cc546ef9e333"
format_stamp: "Formatted at 2026-09-26 06:18:01 on dist-test-slave-hndd"
server_key: "3975f58e155ce13dc1511482e4208a6b"
server_key_iv: "513b5d4d4999168638eefb54169f83e8"
server_key_version: "encryptionkey@0"
I20260926 06:18:01.055099 18187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-2/wal/instance:
uuid: "9ef9321e79e349e4af20cc546ef9e333"
format_stamp: "Formatted at 2026-09-26 06:18:01 on dist-test-slave-hndd"
server_key: "3975f58e155ce13dc1511482e4208a6b"
server_key_iv: "513b5d4d4999168638eefb54169f83e8"
server_key_version: "encryptionkey@0"
I20260926 06:18:01.056481 18187 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:18:01.057277 18202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:18:01.057493 18187 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:18:01.057605 18187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-2/wal
uuid: "9ef9321e79e349e4af20cc546ef9e333"
format_stamp: "Formatted at 2026-09-26 06:18:01 on dist-test-slave-hndd"
server_key: "3975f58e155ce13dc1511482e4208a6b"
server_key_iv: "513b5d4d4999168638eefb54169f83e8"
server_key_version: "encryptionkey@0"
I20260926 06:18:01.057688 18187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:18:01.076699 18187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:18:01.079033 18187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:18:01.079216 18187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:18:01.079464 18187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:18:01.079810 18187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:18:01.079866 18187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:18:01.079917 18187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:18:01.079948 18187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:18:01.086665 18187 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.195:39507
I20260926 06:18:01.086706 18315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.195:39507 every 8 connection(s)
I20260926 06:18:01.087102 18187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
Sep 26 06:18:01 dist-test-slave-hndd krb5kdc[17812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403481, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.195@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:18:01.096048 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 18187
I20260926 06:18:01.096167 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-2/wal/instance
I20260926 06:18:01.096354 2491 external_mini_cluster.cc:1428] Setting key 135fdfa43f76cb17eb7b3ea8ce0aa041
I20260926 06:18:01.099733 17896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:01.087942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:46959 (local address 127.2.110.254:42699)
0926 06:18:01.088018 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 06:18:01.088020 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:18:01.088253 (+ 233us) server_negotiation.cc:407] Connection header received
0926 06:18:01.088771 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:18:01.088772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:18:01.088783 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:18:01.088852 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:18:01.090225 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:01.091091 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:01.092670 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:01.093131 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:01.093230 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:01.096193 (+ 2963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:18:01.096195 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:18:01.096196 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:18:01.096205 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:18:01.098253 (+ 2048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:18:01.098710 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:18:01.098711 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:18:01.098712 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:18:01.098773 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:18:01.099044 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:18:01.099045 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:18:01.099045 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:18:01.099180 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:18:01.099256 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 06:18:01.099541 (+ 285us) server_negotiation.cc:299] Negotiation successful
0926 06:18:01.099633 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 06:18:01.099962 18318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:01.087927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.195:46959)
0926 06:18:01.088092 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:01.088139 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 06:18:01.088686 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:18:01.088906 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:18:01.088910 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:18:01.089757 (+ 847us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:18:01.090100 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:18:01.090106 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:18:01.091213 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:18:01.091214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:18:01.092579 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:18:01.092583 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:18:01.093244 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:18:01.093245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:18:01.093457 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:01.094145 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:18:01.094156 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:18:01.096085 (+ 1929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:18:01.098351 (+ 2266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:18:01.098362 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:18:01.098364 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:18:01.098629 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:18:01.098847 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:18:01.098848 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:18:01.098849 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:18:01.098976 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:18:01.099247 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:18:01.099254 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:18:01.099406 (+ 152us) client_negotiation.cc:769] Sending connection context
0926 06:18:01.099518 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 06:18:01.099695 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":87,"threads_started":1}
I20260926 06:18:01.100461 18316 heartbeater.cc:347] Connected to a master server at 127.2.110.254:42699
I20260926 06:18:01.100548 18316 heartbeater.cc:464] Registering TS with master...
I20260926 06:18:01.100744 18316 heartbeater.cc:511] Master 127.2.110.254:42699 requested a full tablet report, sending...
I20260926 06:18:01.101181 17857 ts_manager.cc:194] Registered new tserver with Master: 9ef9321e79e349e4af20cc546ef9e333 (127.2.110.195:39507)
I20260926 06:18:01.101768 17857 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.195@KRBTEST.COM'} at 127.2.110.195:46959
I20260926 06:18:01.108974 2491 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 06:18:01.117013 17896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:01.110441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42764 (local address 127.2.110.254:42699)
0926 06:18:01.110525 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 06:18:01.110528 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:18:01.110623 (+ 95us) server_negotiation.cc:407] Connection header received
0926 06:18:01.110713 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:18:01.110714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:18:01.110726 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:18:01.110796 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:18:01.111413 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:01.111914 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:01.112644 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:01.112833 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:01.112918 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:01.114272 (+ 1354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:18:01.114274 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:18:01.114274 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:18:01.114281 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:18:01.115770 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:18:01.116152 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:18:01.116153 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:18:01.116154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:18:01.116213 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:18:01.116406 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:18:01.116407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:18:01.116407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:18:01.116570 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:18:01.116689 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 06:18:01.116806 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 06:18:01.116890 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 06:18:01.119009 17857 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42764:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:18:01.120718 17857 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:18:01.128574 18327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:01.123909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.193:41177 (local address 127.0.0.1:34002)
0926 06:18:01.124143 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:01.124187 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 06:18:01.124266 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:18:01.124750 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:18:01.124750 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:18:01.124752 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:18:01.124790 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:18:01.124792 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:18:01.126263 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:18:01.126264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:18:01.127487 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:18:01.127490 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:18:01.128200 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:18:01.128201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:18:01.128283 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:01.128299 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 06:18:01.128403 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 06:18:01.128476 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":59,"threads_started":1}
I20260926 06:18:01.128626 18332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:01.124247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46664 (local address 127.2.110.194:41841)
0926 06:18:01.124407 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 06:18:01.124413 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 06:18:01.124674 (+ 261us) server_negotiation.cc:407] Connection header received
0926 06:18:01.124827 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:18:01.124830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:18:01.124847 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:18:01.124930 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:18:01.125038 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:01.126225 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:01.127425 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:01.128245 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:01.128331 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:01.128390 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 06:18:01.128479 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 06:18:01.128529 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":79,"threads_started":1}
I20260926 06:18:01.128690 18329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:01.123909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.195:39507 (local address 127.0.0.1:56914)
0926 06:18:01.124413 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:01.124426 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:18:01.124497 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:18:01.124703 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:18:01.124706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:18:01.124709 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:18:01.124790 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:18:01.124793 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:18:01.125839 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:18:01.125840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:18:01.127231 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:18:01.127238 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:18:01.128349 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:18:01.128350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:18:01.128453 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:01.128458 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 06:18:01.128498 (+ 40us) client_negotiation.cc:240] Negotiation successful
0926 06:18:01.128548 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"mutex_wait_us":48,"thread_start_us":109,"threads_started":1}
I20260926 06:18:01.128974 18330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:01.124067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56914 (local address 127.2.110.195:39507)
0926 06:18:01.124497 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0926 06:18:01.124501 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:18:01.124512 (+ 11us) server_negotiation.cc:407] Connection header received
0926 06:18:01.124566 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:18:01.124568 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:18:01.124584 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:18:01.124871 (+ 287us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:18:01.124962 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:01.125740 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:01.127477 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:01.128246 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:01.128641 (+ 395us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:01.128837 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0926 06:18:01.128860 (+ 23us) server_negotiation.cc:299] Negotiation successful
0926 06:18:01.128902 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":109,"threads_started":1}
I20260926 06:18:01.128574 18328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:01.123995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.194:41841 (local address 127.0.0.1:46664)
0926 06:18:01.124608 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:01.124618 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:18:01.124704 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:18:01.124936 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:18:01.124936 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:18:01.124938 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:18:01.124972 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:18:01.124973 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:18:01.126315 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:18:01.126316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:18:01.127292 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:18:01.127295 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:18:01.128331 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:18:01.128332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:18:01.128410 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:01.128414 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:18:01.128463 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 06:18:01.128495 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":100,"threads_started":1}
I20260926 06:18:01.129513 18331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:01.124200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34002 (local address 127.2.110.193:41177)
0926 06:18:01.124541 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0926 06:18:01.124547 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 06:18:01.124558 (+ 11us) server_negotiation.cc:407] Connection header received
0926 06:18:01.124607 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:18:01.124610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:18:01.124632 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:18:01.124753 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:18:01.124849 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:01.126151 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:01.127591 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:01.128100 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:01.128200 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:01.128263 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 06:18:01.129370 (+ 1107us) server_negotiation.cc:299] Negotiation successful
0926 06:18:01.129423 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":119,"threads_started":1}
I20260926 06:18:01.129971 18250 tablet_service.cc:1467] Processing CreateTablet for tablet 191d3182fdbc42bfbd2eef9cf349e26e (DEFAULT_TABLE table=test-table [id=6a2ae0020ae1436dbc2e53f94791ab89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:18:01.129971 18116 tablet_service.cc:1467] Processing CreateTablet for tablet 191d3182fdbc42bfbd2eef9cf349e26e (DEFAULT_TABLE table=test-table [id=6a2ae0020ae1436dbc2e53f94791ab89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:18:01.130375 18250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 191d3182fdbc42bfbd2eef9cf349e26e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:18:01.130375 18116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 191d3182fdbc42bfbd2eef9cf349e26e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:18:01.133958 18334 tablet_bootstrap.cc:492] T 191d3182fdbc42bfbd2eef9cf349e26e P 9ef9321e79e349e4af20cc546ef9e333: Bootstrap starting.
I20260926 06:18:01.134132 17982 tablet_service.cc:1467] Processing CreateTablet for tablet 191d3182fdbc42bfbd2eef9cf349e26e (DEFAULT_TABLE table=test-table [id=6a2ae0020ae1436dbc2e53f94791ab89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:18:01.134514 17982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 191d3182fdbc42bfbd2eef9cf349e26e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:18:01.134742 18334 tablet_bootstrap.cc:654] T 191d3182fdbc42bfbd2eef9cf349e26e P 9ef9321e79e349e4af20cc546ef9e333: Neither blocks nor log segments found. Creating new log.
I20260926 06:18:01.135401 18334 log.cc:824] T 191d3182fdbc42bfbd2eef9cf349e26e P 9ef9321e79e349e4af20cc546ef9e333: Log is configured to *not* fsync() on all Append() calls
I20260926 06:18:01.135792 18333 tablet_bootstrap.cc:492] T 191d3182fdbc42bfbd2eef9cf349e26e P 66143d40989641d3a0da9f5e895b22b4: Bootstrap starting.
I20260926 06:18:01.136153 18334 tablet_bootstrap.cc:492] T 191d3182fdbc42bfbd2eef9cf349e26e P 9ef9321e79e349e4af20cc546ef9e333: No bootstrap required, opened a new log
I20260926 06:18:01.136204 18334 ts_tablet_manager.cc:1397] T 191d3182fdbc42bfbd2eef9cf349e26e P 9ef9321e79e349e4af20cc546ef9e333: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:18:01.136518 18333 tablet_bootstrap.cc:654] T 191d3182fdbc42bfbd2eef9cf349e26e P 66143d40989641d3a0da9f5e895b22b4: Neither blocks nor log segments found. Creating new log.
I20260926 06:18:01.137112 18333 log.cc:824] T 191d3182fdbc42bfbd2eef9cf349e26e P 66143d40989641d3a0da9f5e895b22b4: Log is configured to *not* fsync() on all Append() calls
I20260926 06:18:01.137722 18334 raft_consensus.cc:348] T 191d3182fdbc42bfbd2eef9cf349e26e P 9ef9321e79e349e4af20cc546ef9e333 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef9321e79e349e4af20cc546ef9e333" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 39507 } } peers { permanent_uuid: "e559cfe3ebf84f7bb26952adfa617feb" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 41177 } } peers { permanent_uuid: "66143d40989641d3a0da9f5e895b22b4" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 41841 } }
I20260926 06:18:01.137856 18334 raft_consensus.cc:374] T 191d3182fdbc42bfbd2eef9cf349e26e P 9ef9321e79e349e4af20cc546ef9e333 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:18:01.137876 18334 raft_consensus.cc:729] T 191d3182fdbc42bfbd2eef9cf349e26e P 9ef9321e79e349e4af20cc546ef9e333 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef9321e79e349e4af20cc546ef9e333, State: Initialized, Role: FOLLOWER
I20260926 06:18:01.137992 18334 consensus_queue.cc:260] T 191d3182fdbc42bfbd2eef9cf349e26e P 9ef9321e79e349e4af20cc546ef9e333 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef9321e79e349e4af20cc546ef9e333" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 39507 } } peers { permanent_uuid: "e559cfe3ebf84f7bb26952adfa617feb" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 41177 } } peers { permanent_uuid: "66143d40989641d3a0da9f5e895b22b4" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 41841 } }
I20260926 06:18:01.138091 18333 tablet_bootstrap.cc:492] T 191d3182fdbc42bfbd2eef9cf349e26e P 66143d40989641d3a0da9f5e895b22b4: No bootstrap required, opened a new log
I20260926 06:18:01.138154 18333 ts_tablet_manager.cc:1397] T 191d3182fdbc42bfbd2eef9cf349e26e P 66143d40989641d3a0da9f5e895b22b4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 06:18:01.138302 18316 heartbeater.cc:503] Master 127.2.110.254:42699 was elected leader, sending a full tablet report...
I20260926 06:18:01.138330 18334 ts_tablet_manager.cc:1428] T 191d3182fdbc42bfbd2eef9cf349e26e P 9ef9321e79e349e4af20cc546ef9e333: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 06:18:01.138964 18338 tablet_bootstrap.cc:492] T 191d3182fdbc42bfbd2eef9cf349e26e P e559cfe3ebf84f7bb26952adfa617feb: Bootstrap starting.
I20260926 06:18:01.139686 18338 tablet_bootstrap.cc:654] T 191d3182fdbc42bfbd2eef9cf349e26e P e559cfe3ebf84f7bb26952adfa617feb: Neither blocks nor log segments found. Creating new log.
I20260926 06:18:01.140347 18338 log.cc:824] T 191d3182fdbc42bfbd2eef9cf349e26e P e559cfe3ebf84f7bb26952adfa617feb: Log is configured to *not* fsync() on all Append() calls
I20260926 06:18:01.140728 18333 raft_consensus.cc:348] T 191d3182fdbc42bfbd2eef9cf349e26e P 66143d40989641d3a0da9f5e895b22b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef9321e79e349e4af20cc546ef9e333" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 39507 } } peers { permanent_uuid: "e559cfe3ebf84f7bb26952adfa617feb" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 41177 } } peers { permanent_uuid: "66143d40989641d3a0da9f5e895b22b4" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 41841 } }
I20260926 06:18:01.140873 18333 raft_consensus.cc:374] T 191d3182fdbc42bfbd2eef9cf349e26e P 66143d40989641d3a0da9f5e895b22b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:18:01.140898 18333 raft_consensus.cc:729] T 191d3182fdbc42bfbd2eef9cf349e26e P 66143d40989641d3a0da9f5e895b22b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66143d40989641d3a0da9f5e895b22b4, State: Initialized, Role: FOLLOWER
I20260926 06:18:01.141062 18333 consensus_queue.cc:260] T 191d3182fdbc42bfbd2eef9cf349e26e P 66143d40989641d3a0da9f5e895b22b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef9321e79e349e4af20cc546ef9e333" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 39507 } } peers { permanent_uuid: "e559cfe3ebf84f7bb26952adfa617feb" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 41177 } } peers { permanent_uuid: "66143d40989641d3a0da9f5e895b22b4" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 41841 } }
I20260926 06:18:01.141398 18338 tablet_bootstrap.cc:492] T 191d3182fdbc42bfbd2eef9cf349e26e P e559cfe3ebf84f7bb26952adfa617feb: No bootstrap required, opened a new log
I20260926 06:18:01.141413 18182 heartbeater.cc:503] Master 127.2.110.254:42699 was elected leader, sending a full tablet report...
I20260926 06:18:01.141490 18338 ts_tablet_manager.cc:1397] T 191d3182fdbc42bfbd2eef9cf349e26e P e559cfe3ebf84f7bb26952adfa617feb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 06:18:01.141399 18333 ts_tablet_manager.cc:1428] T 191d3182fdbc42bfbd2eef9cf349e26e P 66143d40989641d3a0da9f5e895b22b4: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 06:18:01.143169 18338 raft_consensus.cc:348] T 191d3182fdbc42bfbd2eef9cf349e26e P e559cfe3ebf84f7bb26952adfa617feb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef9321e79e349e4af20cc546ef9e333" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 39507 } } peers { permanent_uuid: "e559cfe3ebf84f7bb26952adfa617feb" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 41177 } } peers { permanent_uuid: "66143d40989641d3a0da9f5e895b22b4" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 41841 } }
I20260926 06:18:01.143311 18338 raft_consensus.cc:374] T 191d3182fdbc42bfbd2eef9cf349e26e P e559cfe3ebf84f7bb26952adfa617feb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:18:01.143333 18338 raft_consensus.cc:729] T 191d3182fdbc42bfbd2eef9cf349e26e P e559cfe3ebf84f7bb26952adfa617feb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e559cfe3ebf84f7bb26952adfa617feb, State: Initialized, Role: FOLLOWER
I20260926 06:18:01.143477 18338 consensus_queue.cc:260] T 191d3182fdbc42bfbd2eef9cf349e26e P e559cfe3ebf84f7bb26952adfa617feb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef9321e79e349e4af20cc546ef9e333" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 39507 } } peers { permanent_uuid: "e559cfe3ebf84f7bb26952adfa617feb" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 41177 } } peers { permanent_uuid: "66143d40989641d3a0da9f5e895b22b4" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 41841 } }
I20260926 06:18:01.143754 18338 ts_tablet_manager.cc:1428] T 191d3182fdbc42bfbd2eef9cf349e26e P e559cfe3ebf84f7bb26952adfa617feb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:18:01.143795 18048 heartbeater.cc:503] Master 127.2.110.254:42699 was elected leader, sending a full tablet report...
W20260926 06:18:01.166157 18183 tablet.cc:2369] T 191d3182fdbc42bfbd2eef9cf349e26e P 66143d40989641d3a0da9f5e895b22b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 06:18:01.258670 18049 tablet.cc:2369] T 191d3182fdbc42bfbd2eef9cf349e26e P e559cfe3ebf84f7bb26952adfa617feb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:18:01.316129 18340 raft_consensus.cc:482] T 191d3182fdbc42bfbd2eef9cf349e26e P 66143d40989641d3a0da9f5e895b22b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:18:01.316329 18340 raft_consensus.cc:504] T 191d3182fdbc42bfbd2eef9cf349e26e P 66143d40989641d3a0da9f5e895b22b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef9321e79e349e4af20cc546ef9e333" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 39507 } } peers { permanent_uuid: "e559cfe3ebf84f7bb26952adfa617feb" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 41177 } } peers { permanent_uuid: "66143d40989641d3a0da9f5e895b22b4" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 41841 } }
I20260926 06:18:01.316761 18340 leader_election.cc:288] T 191d3182fdbc42bfbd2eef9cf349e26e P 66143d40989641d3a0da9f5e895b22b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ef9321e79e349e4af20cc546ef9e333 (127.2.110.195:39507), e559cfe3ebf84f7bb26952adfa617feb (127.2.110.193:41177)
I20260926 06:18:01.319998 18184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:01.316934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.193:41177 (local address 127.2.110.194:40405)
0926 06:18:01.316986 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:01.316999 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:18:01.317049 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:18:01.317266 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:18:01.317268 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:18:01.317270 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:18:01.317374 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:18:01.317377 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:18:01.318059 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:18:01.318061 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:18:01.319004 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:18:01.319012 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:18:01.319679 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:18:01.319680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:18:01.319774 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:01.319778 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:18:01.319848 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 06:18:01.319903 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":17}
I20260926 06:18:01.320071 18331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:01.317064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:40405 (local address 127.2.110.193:41177)
0926 06:18:01.317121 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 06:18:01.317123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:18:01.317133 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:18:01.317163 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:18:01.317164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:18:01.317174 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:18:01.317235 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:18:01.317460 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:01.317975 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:01.319211 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:01.319564 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:01.319638 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:01.319646 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 06:18:01.319909 (+ 263us) server_negotiation.cc:299] Negotiation successful
0926 06:18:01.319958 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 06:18:01.320204 18342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:01.316976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.195:39507 (local address 127.2.110.194:36969)
0926 06:18:01.317174 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:01.317184 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:18:01.317235 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:18:01.317406 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:18:01.317407 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:18:01.317408 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:18:01.317495 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:18:01.317496 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:18:01.318117 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:18:01.318118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:18:01.319032 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:18:01.319035 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:18:01.319929 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:18:01.319931 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:18:01.320000 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:01.320004 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:18:01.320073 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 06:18:01.320118 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":130,"threads_started":1}
I20260926 06:18:01.320251 18330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:01.317082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:36969 (local address 127.2.110.195:39507)
0926 06:18:01.317151 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 06:18:01.317153 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:18:01.317244 (+ 91us) server_negotiation.cc:407] Connection header received
0926 06:18:01.317295 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:18:01.317296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:18:01.317305 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:18:01.317385 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:18:01.317565 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:01.318034 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:01.319210 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:01.319837 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:01.319914 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:01.319924 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 06:18:01.320110 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 06:18:01.320160 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 06:18:01.320547 18002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "191d3182fdbc42bfbd2eef9cf349e26e" candidate_uuid: "66143d40989641d3a0da9f5e895b22b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e559cfe3ebf84f7bb26952adfa617feb" is_pre_election: true
I20260926 06:18:01.320714 18002 raft_consensus.cc:2454] T 191d3182fdbc42bfbd2eef9cf349e26e P e559cfe3ebf84f7bb26952adfa617feb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66143d40989641d3a0da9f5e895b22b4 in term 0.
I20260926 06:18:01.320712 18270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "191d3182fdbc42bfbd2eef9cf349e26e" candidate_uuid: "66143d40989641d3a0da9f5e895b22b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef9321e79e349e4af20cc546ef9e333" is_pre_election: true
I20260926 06:18:01.320865 18270 raft_consensus.cc:2454] T 191d3182fdbc42bfbd2eef9cf349e26e P 9ef9321e79e349e4af20cc546ef9e333 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66143d40989641d3a0da9f5e895b22b4 in term 0.
I20260926 06:18:01.320974 18072 leader_election.cc:302] T 191d3182fdbc42bfbd2eef9cf349e26e P 66143d40989641d3a0da9f5e895b22b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66143d40989641d3a0da9f5e895b22b4, e559cfe3ebf84f7bb26952adfa617feb; no voters:
I20260926 06:18:01.321177 18340 raft_consensus.cc:2781] T 191d3182fdbc42bfbd2eef9cf349e26e P 66143d40989641d3a0da9f5e895b22b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:18:01.321240 18340 raft_consensus.cc:482] T 191d3182fdbc42bfbd2eef9cf349e26e P 66143d40989641d3a0da9f5e895b22b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:18:01.321272 18340 raft_consensus.cc:3037] T 191d3182fdbc42bfbd2eef9cf349e26e P 66143d40989641d3a0da9f5e895b22b4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:18:01.322028 18340 raft_consensus.cc:504] T 191d3182fdbc42bfbd2eef9cf349e26e P 66143d40989641d3a0da9f5e895b22b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef9321e79e349e4af20cc546ef9e333" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 39507 } } peers { permanent_uuid: "e559cfe3ebf84f7bb26952adfa617feb" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 41177 } } peers { permanent_uuid: "66143d40989641d3a0da9f5e895b22b4" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 41841 } }
I20260926 06:18:01.322193 18340 leader_election.cc:288] T 191d3182fdbc42bfbd2eef9cf349e26e P 66143d40989641d3a0da9f5e895b22b4 [CANDIDATE]: Term 1 election: Requested vote from peers 9ef9321e79e349e4af20cc546ef9e333 (127.2.110.195:39507), e559cfe3ebf84f7bb26952adfa617feb (127.2.110.193:41177)
I20260926 06:18:01.322374 18270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "191d3182fdbc42bfbd2eef9cf349e26e" candidate_uuid: "66143d40989641d3a0da9f5e895b22b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef9321e79e349e4af20cc546ef9e333"
I20260926 06:18:01.322448 18270 raft_consensus.cc:3037] T 191d3182fdbc42bfbd2eef9cf349e26e P 9ef9321e79e349e4af20cc546ef9e333 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:18:01.323213 18270 raft_consensus.cc:2454] T 191d3182fdbc42bfbd2eef9cf349e26e P 9ef9321e79e349e4af20cc546ef9e333 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66143d40989641d3a0da9f5e895b22b4 in term 1.
I20260926 06:18:01.323351 18002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "191d3182fdbc42bfbd2eef9cf349e26e" candidate_uuid: "66143d40989641d3a0da9f5e895b22b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e559cfe3ebf84f7bb26952adfa617feb"
I20260926 06:18:01.323436 18070 leader_election.cc:302] T 191d3182fdbc42bfbd2eef9cf349e26e P 66143d40989641d3a0da9f5e895b22b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66143d40989641d3a0da9f5e895b22b4, 9ef9321e79e349e4af20cc546ef9e333; no voters:
I20260926 06:18:01.323437 18002 raft_consensus.cc:3037] T 191d3182fdbc42bfbd2eef9cf349e26e P e559cfe3ebf84f7bb26952adfa617feb [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:18:01.323534 18340 raft_consensus.cc:2781] T 191d3182fdbc42bfbd2eef9cf349e26e P 66143d40989641d3a0da9f5e895b22b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:18:01.323688 18340 raft_consensus.cc:686] T 191d3182fdbc42bfbd2eef9cf349e26e P 66143d40989641d3a0da9f5e895b22b4 [term 1 LEADER]: Becoming Leader. State: Replica: 66143d40989641d3a0da9f5e895b22b4, State: Running, Role: LEADER
I20260926 06:18:01.323796 18340 consensus_queue.cc:237] T 191d3182fdbc42bfbd2eef9cf349e26e P 66143d40989641d3a0da9f5e895b22b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef9321e79e349e4af20cc546ef9e333" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 39507 } } peers { permanent_uuid: "e559cfe3ebf84f7bb26952adfa617feb" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 41177 } } peers { permanent_uuid: "66143d40989641d3a0da9f5e895b22b4" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 41841 } }
I20260926 06:18:01.324313 18002 raft_consensus.cc:2454] T 191d3182fdbc42bfbd2eef9cf349e26e P e559cfe3ebf84f7bb26952adfa617feb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66143d40989641d3a0da9f5e895b22b4 in term 1.
I20260926 06:18:01.324633 17857 catalog_manager.cc:5495] T 191d3182fdbc42bfbd2eef9cf349e26e P 66143d40989641d3a0da9f5e895b22b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66143d40989641d3a0da9f5e895b22b4 (127.2.110.194). New cstate: current_term: 1 leader_uuid: "66143d40989641d3a0da9f5e895b22b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef9321e79e349e4af20cc546ef9e333" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 39507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e559cfe3ebf84f7bb26952adfa617feb" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 41177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66143d40989641d3a0da9f5e895b22b4" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 41841 } health_report { overall_health: HEALTHY } } }
W20260926 06:18:01.337731 18317 tablet.cc:2369] T 191d3182fdbc42bfbd2eef9cf349e26e P 9ef9321e79e349e4af20cc546ef9e333: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:18:01.356498 18332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:01.354073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46678 (local address 127.2.110.194:41841)
0926 06:18:01.354147 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 06:18:01.354149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:18:01.354156 (+ 7us) server_negotiation.cc:407] Connection header received
0926 06:18:01.354193 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:18:01.354194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:18:01.354213 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:18:01.354294 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:18:01.354447 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:01.354935 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:01.355704 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:01.355898 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:01.355974 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:01.356137 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:18:01.356252 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:18:01.356350 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 06:18:01.356369 (+ 19us) server_negotiation.cc:299] Negotiation successful
0926 06:18:01.356391 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 06:18:01.358558 18002 raft_consensus.cc:1261] T 191d3182fdbc42bfbd2eef9cf349e26e P e559cfe3ebf84f7bb26952adfa617feb [term 1 FOLLOWER]: Refusing update from remote peer 66143d40989641d3a0da9f5e895b22b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:18:01.358567 18270 raft_consensus.cc:1261] T 191d3182fdbc42bfbd2eef9cf349e26e P 9ef9321e79e349e4af20cc546ef9e333 [term 1 FOLLOWER]: Refusing update from remote peer 66143d40989641d3a0da9f5e895b22b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:18:01.358949 18340 consensus_queue.cc:1035] T 191d3182fdbc42bfbd2eef9cf349e26e P 66143d40989641d3a0da9f5e895b22b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e559cfe3ebf84f7bb26952adfa617feb" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 41177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:18:01.359091 18343 consensus_queue.cc:1035] T 191d3182fdbc42bfbd2eef9cf349e26e P 66143d40989641d3a0da9f5e895b22b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ef9321e79e349e4af20cc546ef9e333" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 39507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:18:01.364662 17855 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:18:01.364768 17855 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:18:01.365726 17855 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc: Sending DeleteTablet for 3 replicas of tablet 191d3182fdbc42bfbd2eef9cf349e26e
I20260926 06:18:01.366125 18116 tablet_service.cc:1514] Processing DeleteTablet for tablet 191d3182fdbc42bfbd2eef9cf349e26e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:18:01 UTC) from {username='kudu', principal='kudu/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:46664
I20260926 06:18:01.366154 17982 tablet_service.cc:1514] Processing DeleteTablet for tablet 191d3182fdbc42bfbd2eef9cf349e26e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:18:01 UTC) from {username='kudu', principal='kudu/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:34002
I20260926 06:18:01.366205 18250 tablet_service.cc:1514] Processing DeleteTablet for tablet 191d3182fdbc42bfbd2eef9cf349e26e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:18:01 UTC) from {username='kudu', principal='kudu/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:56914
I20260926 06:18:01.366439 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 17919
I20260926 06:18:01.366598 18356 tablet_replica.cc:330] T 191d3182fdbc42bfbd2eef9cf349e26e P 66143d40989641d3a0da9f5e895b22b4: stopping tablet replica
I20260926 06:18:01.366743 18358 tablet_replica.cc:330] T 191d3182fdbc42bfbd2eef9cf349e26e P 9ef9321e79e349e4af20cc546ef9e333: stopping tablet replica
I20260926 06:18:01.366914 18358 raft_consensus.cc:2229] T 191d3182fdbc42bfbd2eef9cf349e26e P 9ef9321e79e349e4af20cc546ef9e333 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:18:01.367079 18356 raft_consensus.cc:2229] T 191d3182fdbc42bfbd2eef9cf349e26e P 66143d40989641d3a0da9f5e895b22b4 [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:18:01.367249 18358 raft_consensus.cc:2258] T 191d3182fdbc42bfbd2eef9cf349e26e P 9ef9321e79e349e4af20cc546ef9e333 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:18:01.367409 18356 raft_consensus.cc:2258] T 191d3182fdbc42bfbd2eef9cf349e26e P 66143d40989641d3a0da9f5e895b22b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:18:01.367952 18358 ts_tablet_manager.cc:1905] T 191d3182fdbc42bfbd2eef9cf349e26e P 9ef9321e79e349e4af20cc546ef9e333: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:18:01.368119 18356 ts_tablet_manager.cc:1905] T 191d3182fdbc42bfbd2eef9cf349e26e P 66143d40989641d3a0da9f5e895b22b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:18:01.369840 18358 ts_tablet_manager.cc:1918] T 191d3182fdbc42bfbd2eef9cf349e26e P 9ef9321e79e349e4af20cc546ef9e333: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:18:01.370002 18358 log.cc:1196] T 191d3182fdbc42bfbd2eef9cf349e26e P 9ef9321e79e349e4af20cc546ef9e333: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-2/wal/wals/191d3182fdbc42bfbd2eef9cf349e26e
I20260926 06:18:01.370476 18356 ts_tablet_manager.cc:1918] T 191d3182fdbc42bfbd2eef9cf349e26e P 66143d40989641d3a0da9f5e895b22b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:18:01.370602 18356 log.cc:1196] T 191d3182fdbc42bfbd2eef9cf349e26e P 66143d40989641d3a0da9f5e895b22b4: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-1/wal/wals/191d3182fdbc42bfbd2eef9cf349e26e
I20260926 06:18:01.370388 18358 ts_tablet_manager.cc:1939] T 191d3182fdbc42bfbd2eef9cf349e26e P 9ef9321e79e349e4af20cc546ef9e333: Deleting consensus metadata
I20260926 06:18:01.370920 18356 ts_tablet_manager.cc:1939] T 191d3182fdbc42bfbd2eef9cf349e26e P 66143d40989641d3a0da9f5e895b22b4: Deleting consensus metadata
I20260926 06:18:01.371461 17841 catalog_manager.cc:4841] TS 66143d40989641d3a0da9f5e895b22b4 (127.2.110.194:41841): tablet 191d3182fdbc42bfbd2eef9cf349e26e (table test-table [id=6a2ae0020ae1436dbc2e53f94791ab89]) successfully deleted
I20260926 06:18:01.371601 17842 catalog_manager.cc:4841] TS 9ef9321e79e349e4af20cc546ef9e333 (127.2.110.195:39507): tablet 191d3182fdbc42bfbd2eef9cf349e26e (table test-table [id=6a2ae0020ae1436dbc2e53f94791ab89]) successfully deleted
W20260926 06:18:01.374188 17844 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.110.193:41177 (error 108)
I20260926 06:18:01.374946 18328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:01.374795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.193:41177 (local address 127.0.0.1:34014)
0926 06:18:01.374861 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:01.374907 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.193:41177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 06:18:01.374984 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 18053
W20260926 06:18:01.375087 17844 catalog_manager.cc:4584] TS e559cfe3ebf84f7bb26952adfa617feb (127.2.110.193:41177): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 191d3182fdbc42bfbd2eef9cf349e26e: Network error: Client connection negotiation failed: client connection to 127.2.110.193:41177: connect: Connection refused (error 111)
I20260926 06:18:01.380182 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 18187
I20260926 06:18:01.385051 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 17825
I20260926 06:18:21.390890 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:42699
--webserver_interface=127.2.110.254
--webserver_port=40105
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:36009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.110.254:42699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:18:21.474169 18360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:18:21.474386 18360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:18:21.474412 18360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:18:21.475524 18360 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 06:18:21.475564 18360 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 06:18:21.475579 18360 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 06:18:21.475590 18360 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:18:21.475603 18360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:18:21.475616 18360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:18:21.475629 18360 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:18:21.475641 18360 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:18:21.477078 18360 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:36009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.110.254:42699
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:42699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.110.254
--webserver_port=40105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:18:21.477295 18360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:18:21.477569 18360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:18:21.486435 18369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:18:21.486431 18366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:18:21.486431 18367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:18:21.487555 18360 server_base.cc:1034] running on GCE node
I20260926 06:18:21.487864 18360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:18:21.488209 18360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:18:21.489444 18360 hybrid_clock.cc:648] HybridClock initialized: now 1790403501489419 us; error 49 us; skew 500 ppm
Sep 26 06:18:21 dist-test-slave-hndd krb5kdc[17812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403501, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:18:21.492187 18360 init.cc:377] Logged in from keytab as kudu/127.2.110.254@KRBTEST.COM (short username kudu)
I20260926 06:18:21.492708 18360 webserver.cc:460] Webserver started at http://127.2.110.254:40105/ using document root <none> and password file <none>
I20260926 06:18:21.492971 18360 fs_manager.cc:359] Metadata directory not provided
I20260926 06:18:21.493049 18360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:18:21.494675 18360 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:18:21.495604 18376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:18:21.495913 18360 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:18:21.496037 18360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "c8235472aa12413abcb030e25ed11bfc"
format_stamp: "Formatted at 2026-09-26 06:18:00 on dist-test-slave-hndd"
server_key: "272bd0b7ba819a640db60dc1f473c9f6"
server_key_iv: "e3ddca0877505d1ddd96d5ad5f7f11c4"
server_key_version: "encryptionkey@0"
I20260926 06:18:21.496500 18360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:18:21.520196 18360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:18:21.522495 18360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:18:21.522677 18360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:18:21.527731 18360 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.254:42699
I20260926 06:18:21.527805 18428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.254:42699 every 8 connection(s)
I20260926 06:18:21.528223 18360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:18:21.529852 18429 sys_catalog.cc:263] Verifying existing consensus state
I20260926 06:18:21.530454 18429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc: Bootstrap starting.
I20260926 06:18:21.536896 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 18360
I20260926 06:18:21.537326 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.193:41177
--local_ip_for_outbound_sockets=127.2.110.193
--tserver_master_addrs=127.2.110.254:42699
--webserver_port=42771
--webserver_interface=127.2.110.193
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab
--principal=kudu/127.2.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:36009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 06:18:21.539183 18429 log.cc:824] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc: Log is configured to *not* fsync() on all Append() calls
I20260926 06:18:21.541311 18429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 06:18:21.541713 18429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc: Bootstrap complete.
I20260926 06:18:21.544121 18429 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8235472aa12413abcb030e25ed11bfc" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 42699 } }
I20260926 06:18:21.544404 18429 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8235472aa12413abcb030e25ed11bfc, State: Initialized, Role: FOLLOWER
I20260926 06:18:21.544620 18429 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8235472aa12413abcb030e25ed11bfc" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 42699 } }
I20260926 06:18:21.544723 18429 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:18:21.544790 18429 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:18:21.544842 18429 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc [term 1 FOLLOWER]: Advancing to term 2
I20260926 06:18:21.546011 18429 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8235472aa12413abcb030e25ed11bfc" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 42699 } }
I20260926 06:18:21.546145 18429 leader_election.cc:302] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8235472aa12413abcb030e25ed11bfc; no voters:
I20260926 06:18:21.546360 18429 leader_election.cc:288] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 06:18:21.546404 18433 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc [term 2 FOLLOWER]: Leader election won for term 2
I20260926 06:18:21.546521 18433 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc [term 2 LEADER]: Becoming Leader. State: Replica: c8235472aa12413abcb030e25ed11bfc, State: Running, Role: LEADER
I20260926 06:18:21.546649 18429 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:18:21.546665 18433 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8235472aa12413abcb030e25ed11bfc" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 42699 } }
I20260926 06:18:21.547079 18434 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c8235472aa12413abcb030e25ed11bfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8235472aa12413abcb030e25ed11bfc" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 42699 } } }
I20260926 06:18:21.547159 18434 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc [sys.catalog]: This master's current role is: LEADER
I20260926 06:18:21.547827 18435 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8235472aa12413abcb030e25ed11bfc. Latest consensus state: current_term: 2 leader_uuid: "c8235472aa12413abcb030e25ed11bfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8235472aa12413abcb030e25ed11bfc" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 42699 } } }
I20260926 06:18:21.547916 18435 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc [sys.catalog]: This master's current role is: LEADER
I20260926 06:18:21.548141 18441 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:18:21.549018 18441 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:18:21.549134 18441 catalog_manager.cc:1259] Loaded cluster ID: 8987a235b02f488fa2d72089fdb46881
I20260926 06:18:21.549157 18441 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:18:21.550417 18441 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:18:21.550601 18441 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc: Loaded TSK: 0
I20260926 06:18:21.564148 18441 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc: Generated new TSK 1
I20260926 06:18:21.564258 18441 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 06:18:21.633554 18431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:18:21.633774 18431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:18:21.633803 18431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:18:21.635093 18431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:18:21.635144 18431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:18:21.635176 18431 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.193
I20260926 06:18:21.636745 18431 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:36009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab
--principal=kudu/127.2.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.193:41177
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.110.193
--webserver_port=42771
--tserver_master_addrs=127.2.110.254:42699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.193
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:18:21.637032 18431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:18:21.637403 18431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:18:21.642827 18457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:18:21.642863 18460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:18:21.643080 18431 server_base.cc:1034] running on GCE node
W20260926 06:18:21.642863 18458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:18:21.643558 18431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:18:21.643846 18431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:18:21.645038 18431 hybrid_clock.cc:648] HybridClock initialized: now 1790403501645014 us; error 43 us; skew 500 ppm
Sep 26 06:18:21 dist-test-slave-hndd krb5kdc[17812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403501, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:18:21.647977 18431 init.cc:377] Logged in from keytab as kudu/127.2.110.193@KRBTEST.COM (short username kudu)
I20260926 06:18:21.648515 18431 webserver.cc:460] Webserver started at http://127.2.110.193:42771/ using document root <none> and password file <none>
I20260926 06:18:21.648829 18431 fs_manager.cc:359] Metadata directory not provided
I20260926 06:18:21.648917 18431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:18:21.650382 18431 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:18:21.651124 18467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:18:21.651376 18431 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:18:21.651480 18431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-0/wal
uuid: "e559cfe3ebf84f7bb26952adfa617feb"
format_stamp: "Formatted at 2026-09-26 06:18:00 on dist-test-slave-hndd"
server_key: "eeea97a697644d4849d390d10ada85c6"
server_key_iv: "463f9f6720b364bfe68e018124e93a7f"
server_key_version: "encryptionkey@0"
I20260926 06:18:21.651940 18431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:18:21.670707 18431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:18:21.673432 18431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:18:21.673982 18431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:18:21.674315 18431 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:18:21.674907 18474 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 06:18:21.676452 18431 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 06:18:21.676609 18431 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 06:18:21.676750 18431 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 06:18:21.677846 18431 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 06:18:21.677902 18431 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.001s
I20260926 06:18:21.678001 18474 tablet_bootstrap.cc:492] T 191d3182fdbc42bfbd2eef9cf349e26e P e559cfe3ebf84f7bb26952adfa617feb: Bootstrap starting.
I20260926 06:18:21.685158 18431 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.193:41177
I20260926 06:18:21.685456 18581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.193:41177 every 8 connection(s)
I20260926 06:18:21.685694 18431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
I20260926 06:18:21.689329 18474 log.cc:824] T 191d3182fdbc42bfbd2eef9cf349e26e P e559cfe3ebf84f7bb26952adfa617feb: Log is configured to *not* fsync() on all Append() calls
I20260926 06:18:21.692077 18474 tablet_bootstrap.cc:492] T 191d3182fdbc42bfbd2eef9cf349e26e P e559cfe3ebf84f7bb26952adfa617feb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 06:18:21.692420 18474 tablet_bootstrap.cc:492] T 191d3182fdbc42bfbd2eef9cf349e26e P e559cfe3ebf84f7bb26952adfa617feb: Bootstrap complete.
I20260926 06:18:21.693455 18474 ts_tablet_manager.cc:1397] T 191d3182fdbc42bfbd2eef9cf349e26e P e559cfe3ebf84f7bb26952adfa617feb: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.012s
I20260926 06:18:21.694264 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 18431
I20260926 06:18:21.694690 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.194:41841
--local_ip_for_outbound_sockets=127.2.110.194
--tserver_master_addrs=127.2.110.254:42699
--webserver_port=46171
--webserver_interface=127.2.110.194
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab
--principal=kudu/127.2.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:36009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 06:18:21 dist-test-slave-hndd krb5kdc[17812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403501, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.193@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:18:21.696844 18474 raft_consensus.cc:348] T 191d3182fdbc42bfbd2eef9cf349e26e P e559cfe3ebf84f7bb26952adfa617feb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef9321e79e349e4af20cc546ef9e333" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 39507 } } peers { permanent_uuid: "e559cfe3ebf84f7bb26952adfa617feb" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 41177 } } peers { permanent_uuid: "66143d40989641d3a0da9f5e895b22b4" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 41841 } }
I20260926 06:18:21.697186 18474 raft_consensus.cc:729] T 191d3182fdbc42bfbd2eef9cf349e26e P e559cfe3ebf84f7bb26952adfa617feb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e559cfe3ebf84f7bb26952adfa617feb, State: Initialized, Role: FOLLOWER
I20260926 06:18:21.697407 18474 consensus_queue.cc:260] T 191d3182fdbc42bfbd2eef9cf349e26e P e559cfe3ebf84f7bb26952adfa617feb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef9321e79e349e4af20cc546ef9e333" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 39507 } } peers { permanent_uuid: "e559cfe3ebf84f7bb26952adfa617feb" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 41177 } } peers { permanent_uuid: "66143d40989641d3a0da9f5e895b22b4" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 41841 } }
I20260926 06:18:21.697763 18474 ts_tablet_manager.cc:1428] T 191d3182fdbc42bfbd2eef9cf349e26e P e559cfe3ebf84f7bb26952adfa617feb: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260926 06:18:21.701700 18584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:21.686390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.193:51803)
0926 06:18:21.686600 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:21.686647 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 06:18:21.687245 (+ 598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:18:21.689440 (+ 2195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:18:21.689445 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:18:21.689843 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:18:21.690126 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:18:21.690133 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:18:21.691884 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:18:21.691885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:18:21.693260 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:18:21.693265 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:18:21.694100 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:18:21.694101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:18:21.694248 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:21.695133 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:18:21.695149 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:18:21.697600 (+ 2451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:18:21.699637 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:18:21.699645 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:18:21.699647 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:18:21.700001 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:18:21.700335 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:18:21.700335 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:18:21.700336 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:18:21.700486 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:18:21.700996 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:18:21.701003 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:18:21.701186 (+ 183us) client_negotiation.cc:769] Sending connection context
0926 06:18:21.701288 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 06:18:21.701503 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":117,"threads_started":1}
I20260926 06:18:21.701885 18585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:21.686524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:51803 (local address 127.2.110.254:42699)
0926 06:18:21.688890 (+ 2366us) server_negotiation.cc:206] Beginning negotiation
0926 06:18:21.688902 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 06:18:21.688951 (+ 49us) server_negotiation.cc:407] Connection header received
0926 06:18:21.689098 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:18:21.689106 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:18:21.689175 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:18:21.689373 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:18:21.690275 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:21.691611 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:21.693395 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:21.693977 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:21.694094 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:21.697808 (+ 3714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:18:21.697812 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:18:21.697812 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:18:21.697829 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:18:21.699537 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:18:21.700118 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:18:21.700122 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:18:21.700122 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:18:21.700253 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:18:21.700571 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:18:21.700572 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:18:21.700572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:18:21.700910 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:18:21.701015 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 06:18:21.701491 (+ 476us) server_negotiation.cc:299] Negotiation successful
0926 06:18:21.701652 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2304,"thread_start_us":93,"threads_started":1}
I20260926 06:18:21.703512 18582 heartbeater.cc:347] Connected to a master server at 127.2.110.254:42699
I20260926 06:18:21.703686 18582 heartbeater.cc:464] Registering TS with master...
I20260926 06:18:21.704066 18582 heartbeater.cc:511] Master 127.2.110.254:42699 requested a full tablet report, sending...
I20260926 06:18:21.705016 18393 ts_manager.cc:194] Registered new tserver with Master: e559cfe3ebf84f7bb26952adfa617feb (127.2.110.193:41177)
I20260926 06:18:21.706207 18393 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.193@KRBTEST.COM'} at 127.2.110.193:51803
I20260926 06:18:21.707326 18582 heartbeater.cc:503] Master 127.2.110.254:42699 was elected leader, sending a full tablet report...
Sep 26 06:18:21 dist-test-slave-hndd krb5kdc[17812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403501, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.254@KRBTEST.COM for kudu/127.2.110.193@KRBTEST.COM
I20260926 06:18:21.714231 18589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:21.705768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.193:41177 (local address 127.0.0.1:59158)
0926 06:18:21.706087 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:21.706125 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 06:18:21.706216 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:18:21.706427 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:18:21.706432 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:18:21.706784 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:18:21.706834 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:18:21.706837 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:18:21.707882 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:18:21.707883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:18:21.708464 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:18:21.708466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:18:21.709029 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:18:21.709030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:18:21.709117 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:21.709689 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:18:21.709694 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:18:21.711109 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:18:21.712927 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:18:21.712935 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:18:21.712937 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:18:21.713150 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:18:21.713426 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:18:21.713426 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:18:21.713427 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:18:21.713493 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:18:21.713819 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:18:21.713826 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:18:21.713881 (+ 55us) client_negotiation.cc:769] Sending connection context
0926 06:18:21.713980 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 06:18:21.714122 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":112,"threads_started":1}
I20260926 06:18:21.714231 18590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:21.705848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59158 (local address 127.2.110.193:41177)
0926 06:18:21.706020 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 06:18:21.706025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:18:21.706220 (+ 195us) server_negotiation.cc:407] Connection header received
0926 06:18:21.706305 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:18:21.706308 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:18:21.706327 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:18:21.706413 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:18:21.706925 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:21.707798 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:21.708556 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:21.708927 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:21.709027 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:21.711207 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:18:21.711216 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:18:21.711216 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:18:21.711224 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:18:21.712820 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:18:21.713239 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:18:21.713242 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:18:21.713243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:18:21.713308 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:18:21.713562 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:18:21.713563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:18:21.713563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:18:21.713733 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:18:21.713829 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 06:18:21.714030 (+ 201us) server_negotiation.cc:299] Negotiation successful
0926 06:18:21.714124 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":93,"threads_started":1}
I20260926 06:18:21.714658 18516 tablet_service.cc:1514] Processing DeleteTablet for tablet 191d3182fdbc42bfbd2eef9cf349e26e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:18:01 UTC) from {username='kudu', principal='kudu/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:59158
I20260926 06:18:21.715179 18591 tablet_replica.cc:330] stopping tablet replica
I20260926 06:18:21.715303 18591 raft_consensus.cc:2229] T 191d3182fdbc42bfbd2eef9cf349e26e P e559cfe3ebf84f7bb26952adfa617feb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:18:21.715375 18591 raft_consensus.cc:2258] T 191d3182fdbc42bfbd2eef9cf349e26e P e559cfe3ebf84f7bb26952adfa617feb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:18:21.715703 18591 ts_tablet_manager.cc:1905] T 191d3182fdbc42bfbd2eef9cf349e26e P e559cfe3ebf84f7bb26952adfa617feb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:18:21.717835 18591 ts_tablet_manager.cc:1918] T 191d3182fdbc42bfbd2eef9cf349e26e P e559cfe3ebf84f7bb26952adfa617feb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:18:21.717901 18591 log.cc:1196] T 191d3182fdbc42bfbd2eef9cf349e26e P e559cfe3ebf84f7bb26952adfa617feb: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-0/wal/wals/191d3182fdbc42bfbd2eef9cf349e26e
I20260926 06:18:21.718161 18591 ts_tablet_manager.cc:1939] T 191d3182fdbc42bfbd2eef9cf349e26e P e559cfe3ebf84f7bb26952adfa617feb: Deleting consensus metadata
I20260926 06:18:21.718703 18380 catalog_manager.cc:4841] TS e559cfe3ebf84f7bb26952adfa617feb (127.2.110.193:41177): tablet 191d3182fdbc42bfbd2eef9cf349e26e (table test-table [id=6a2ae0020ae1436dbc2e53f94791ab89]) successfully deleted
W20260926 06:18:21.789562 18587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:18:21.789852 18587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:18:21.789896 18587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:18:21.791117 18587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:18:21.791213 18587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:18:21.791266 18587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.194
I20260926 06:18:21.793041 18587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:36009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab
--principal=kudu/127.2.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.194:41841
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.110.194
--webserver_port=46171
--tserver_master_addrs=127.2.110.254:42699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.194
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:18:21.793421 18587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:18:21.793704 18587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:18:21.799228 18599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:18:21.799117 18597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:18:21.799113 18596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:18:21.799626 18587 server_base.cc:1034] running on GCE node
I20260926 06:18:21.799860 18587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:18:21.800163 18587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:18:21.801373 18587 hybrid_clock.cc:648] HybridClock initialized: now 1790403501801318 us; error 30 us; skew 500 ppm
Sep 26 06:18:21 dist-test-slave-hndd krb5kdc[17812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403501, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:18:21.804067 18587 init.cc:377] Logged in from keytab as kudu/127.2.110.194@KRBTEST.COM (short username kudu)
I20260926 06:18:21.805032 18587 webserver.cc:460] Webserver started at http://127.2.110.194:46171/ using document root <none> and password file <none>
I20260926 06:18:21.805310 18587 fs_manager.cc:359] Metadata directory not provided
I20260926 06:18:21.805466 18587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:18:21.807651 18587 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:18:21.808660 18606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:18:21.808938 18587 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:18:21.809072 18587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-1/wal
uuid: "66143d40989641d3a0da9f5e895b22b4"
format_stamp: "Formatted at 2026-09-26 06:18:00 on dist-test-slave-hndd"
server_key: "ba5867f04f8dca91116f12cbf7de80f8"
server_key_iv: "bbcc74d120502733fe96a7a1eff028d3"
server_key_version: "encryptionkey@0"
I20260926 06:18:21.809679 18587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:18:21.820091 18587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:18:21.822577 18587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:18:21.822706 18587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:18:21.822908 18587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:18:21.823264 18587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:18:21.823318 18587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:18:21.823342 18587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:18:21.823356 18587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:18:21.829239 18587 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.194:41841
I20260926 06:18:21.829378 18719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.194:41841 every 8 connection(s)
I20260926 06:18:21.829924 18587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
I20260926 06:18:21.835673 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 18587
I20260926 06:18:21.836031 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.195:39507
--local_ip_for_outbound_sockets=127.2.110.195
--tserver_master_addrs=127.2.110.254:42699
--webserver_port=40435
--webserver_interface=127.2.110.195
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab
--principal=kudu/127.2.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:36009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 06:18:21 dist-test-slave-hndd krb5kdc[17812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403501, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.194@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:18:21.843056 18722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:21.830822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.194:37719)
0926 06:18:21.831088 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:21.831146 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 06:18:21.831824 (+ 678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:18:21.832195 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:18:21.832206 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:18:21.833135 (+ 929us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:18:21.833481 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:18:21.833491 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:18:21.834559 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:18:21.834560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:18:21.835673 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:18:21.835679 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:18:21.836267 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:18:21.836268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:18:21.836404 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:21.837025 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:18:21.837034 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:18:21.839319 (+ 2285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:18:21.841562 (+ 2243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:18:21.841570 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:18:21.841572 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:18:21.841827 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:18:21.842056 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:18:21.842057 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:18:21.842057 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:18:21.842161 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:18:21.842489 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:18:21.842494 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:18:21.842638 (+ 144us) client_negotiation.cc:769] Sending connection context
0926 06:18:21.842740 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 06:18:21.842877 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":133,"threads_started":1}
I20260926 06:18:21.843056 18585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:21.830862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:37719 (local address 127.2.110.254:42699)
0926 06:18:21.830962 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0926 06:18:21.830964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:18:21.831277 (+ 313us) server_negotiation.cc:407] Connection header received
0926 06:18:21.831993 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:18:21.831995 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:18:21.832016 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:18:21.832094 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:18:21.833647 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:21.834386 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:21.835822 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:21.836089 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:21.836182 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:21.839459 (+ 3277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:18:21.839461 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:18:21.839462 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:18:21.839479 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:18:21.841469 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:18:21.841912 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:18:21.841913 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:18:21.841914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:18:21.841984 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:18:21.842232 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:18:21.842233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:18:21.842233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:18:21.842406 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:18:21.842493 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 06:18:21.842798 (+ 305us) server_negotiation.cc:299] Negotiation successful
0926 06:18:21.842912 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 06:18:21.843740 18720 heartbeater.cc:347] Connected to a master server at 127.2.110.254:42699
I20260926 06:18:21.843874 18720 heartbeater.cc:464] Registering TS with master...
I20260926 06:18:21.844106 18720 heartbeater.cc:511] Master 127.2.110.254:42699 requested a full tablet report, sending...
I20260926 06:18:21.844563 18393 ts_manager.cc:194] Registered new tserver with Master: 66143d40989641d3a0da9f5e895b22b4 (127.2.110.194:41841)
I20260926 06:18:21.844957 18393 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.194@KRBTEST.COM'} at 127.2.110.194:37719
W20260926 06:18:21.929850 18723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:18:21.930068 18723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:18:21.930094 18723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:18:21.931301 18723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:18:21.931347 18723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:18:21.931375 18723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.195
I20260926 06:18:21.933051 18723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:36009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab
--principal=kudu/127.2.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.195:39507
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.110.195
--webserver_port=40435
--tserver_master_addrs=127.2.110.254:42699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.195
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:18:21.933573 18723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:18:21.934013 18723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:18:21.939580 18729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:18:21.939841 18723 server_base.cc:1034] running on GCE node
W20260926 06:18:21.939679 18731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:18:21.940033 18728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:18:21.940281 18723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:18:21.940518 18723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:18:21.941665 18723 hybrid_clock.cc:648] HybridClock initialized: now 1790403501941651 us; error 35 us; skew 500 ppm
Sep 26 06:18:21 dist-test-slave-hndd krb5kdc[17812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403501, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:18:21.944478 18723 init.cc:377] Logged in from keytab as kudu/127.2.110.195@KRBTEST.COM (short username kudu)
I20260926 06:18:21.945252 18723 webserver.cc:460] Webserver started at http://127.2.110.195:40435/ using document root <none> and password file <none>
I20260926 06:18:21.945578 18723 fs_manager.cc:359] Metadata directory not provided
I20260926 06:18:21.945681 18723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:18:21.947738 18723 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:18:21.948709 18738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:18:21.949019 18723 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:18:21.949160 18723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-2/wal
uuid: "9ef9321e79e349e4af20cc546ef9e333"
format_stamp: "Formatted at 2026-09-26 06:18:01 on dist-test-slave-hndd"
server_key: "3975f58e155ce13dc1511482e4208a6b"
server_key_iv: "513b5d4d4999168638eefb54169f83e8"
server_key_version: "encryptionkey@0"
I20260926 06:18:21.949630 18723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:18:21.961392 18723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:18:21.964108 18723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:18:21.964306 18723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:18:21.964601 18723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:18:21.964964 18723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:18:21.965026 18723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:18:21.965065 18723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:18:21.965101 18723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:18:21.971673 18723 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.195:39507
I20260926 06:18:21.971726 18851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.195:39507 every 8 connection(s)
I20260926 06:18:21.972105 18723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
I20260926 06:18:21.972748 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 18723
Sep 26 06:18:21 dist-test-slave-hndd krb5kdc[17812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403501, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.195@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:18:21.980520 18855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:21.973164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43118 (local address 127.2.110.254:42699)
0926 06:18:21.973273 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 06:18:21.973275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:18:21.973400 (+ 125us) server_negotiation.cc:407] Connection header received
0926 06:18:21.973494 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:18:21.973494 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:18:21.973504 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:18:21.973557 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:18:21.974368 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:21.974863 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:21.975720 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:21.975933 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:21.976013 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:21.977309 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:18:21.977310 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:18:21.977310 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:18:21.977318 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:18:21.978912 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:18:21.979269 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:18:21.979270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:18:21.979271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:18:21.979337 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:18:21.979895 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:18:21.979896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:18:21.979896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:18:21.980006 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:18:21.980181 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 06:18:21.980277 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 06:18:21.980371 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":54,"threads_started":1}
I20260926 06:18:21.982971 18585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:21.972858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:37713 (local address 127.2.110.254:42699)
0926 06:18:21.972929 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 06:18:21.972931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:18:21.973170 (+ 239us) server_negotiation.cc:407] Connection header received
0926 06:18:21.973929 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:18:21.973929 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:18:21.973935 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:18:21.973966 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:18:21.975089 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:21.975694 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:21.977243 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:21.977562 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:21.977655 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:21.979908 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:18:21.979909 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:18:21.979909 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:18:21.979915 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:18:21.981819 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:18:21.982219 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:18:21.982220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:18:21.982220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:18:21.982269 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:18:21.982453 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:18:21.982453 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:18:21.982454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:18:21.982552 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:18:21.982602 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 06:18:21.982849 (+ 247us) server_negotiation.cc:299] Negotiation successful
0926 06:18:21.982910 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 06:18:21.983172 18854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:21.972871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.195:37713)
0926 06:18:21.973041 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:21.973091 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 06:18:21.973827 (+ 736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:18:21.974064 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:18:21.974069 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:18:21.974757 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:18:21.975001 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:18:21.975007 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:18:21.975947 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:18:21.975949 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:18:21.977077 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:18:21.977083 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:18:21.977668 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:18:21.977669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:18:21.977816 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:21.978275 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:18:21.978283 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:18:21.979806 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:18:21.981914 (+ 2108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:18:21.981920 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:18:21.981922 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:18:21.982153 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:18:21.982314 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:18:21.982314 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:18:21.982315 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:18:21.982400 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:18:21.982610 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:18:21.982615 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:18:21.982736 (+ 121us) client_negotiation.cc:769] Sending connection context
0926 06:18:21.982822 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 06:18:21.982947 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":100,"threads_started":1}
I20260926 06:18:21.983692 18852 heartbeater.cc:347] Connected to a master server at 127.2.110.254:42699
I20260926 06:18:21.983793 18852 heartbeater.cc:464] Registering TS with master...
I20260926 06:18:21.983939 18852 heartbeater.cc:511] Master 127.2.110.254:42699 requested a full tablet report, sending...
I20260926 06:18:21.984308 18393 ts_manager.cc:194] Registered new tserver with Master: 9ef9321e79e349e4af20cc546ef9e333 (127.2.110.195:39507)
I20260926 06:18:21.984876 18393 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.195@KRBTEST.COM'} at 127.2.110.195:37713
I20260926 06:18:21.992669 2491 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 06:18:21.995786 18585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:21.993119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43130 (local address 127.2.110.254:42699)
0926 06:18:21.993184 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 06:18:21.993186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:18:21.993294 (+ 108us) server_negotiation.cc:407] Connection header received
0926 06:18:21.993393 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:18:21.993394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:18:21.993442 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:18:21.993511 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:18:21.993642 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:21.994140 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:21.994930 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:21.995147 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:21.995218 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:21.995416 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:18:21.995536 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:18:21.995640 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 06:18:21.995680 (+ 40us) server_negotiation.cc:299] Negotiation successful
0926 06:18:21.995700 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 06:18:21.996081 18393 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:43130:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:18:21.998601 18393 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:18:22.000816 18516 tablet_service.cc:1467] Processing CreateTablet for tablet 5ec03b6813aa499d8e035917ba6ea022 (DEFAULT_TABLE table=test-table [id=9b6bb26605b64d2aaa71b7884ff41472]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:18:22.001026 18516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ec03b6813aa499d8e035917ba6ea022. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:18:22.002604 18474 tablet_bootstrap.cc:492] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb: Bootstrap starting.
I20260926 06:18:22.002951 18474 tablet_bootstrap.cc:654] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb: Neither blocks nor log segments found. Creating new log.
I20260926 06:18:22.003481 18474 tablet_bootstrap.cc:492] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb: No bootstrap required, opened a new log
I20260926 06:18:22.003525 18474 ts_tablet_manager.cc:1397] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 06:18:22.003865 18474 raft_consensus.cc:348] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef9321e79e349e4af20cc546ef9e333" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 39507 } } peers { permanent_uuid: "66143d40989641d3a0da9f5e895b22b4" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 41841 } } peers { permanent_uuid: "e559cfe3ebf84f7bb26952adfa617feb" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 41177 } }
I20260926 06:18:22.003929 18474 raft_consensus.cc:374] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:18:22.003955 18474 raft_consensus.cc:729] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e559cfe3ebf84f7bb26952adfa617feb, State: Initialized, Role: FOLLOWER
I20260926 06:18:22.004026 18474 consensus_queue.cc:260] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef9321e79e349e4af20cc546ef9e333" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 39507 } } peers { permanent_uuid: "66143d40989641d3a0da9f5e895b22b4" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 41841 } } peers { permanent_uuid: "e559cfe3ebf84f7bb26952adfa617feb" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 41177 } }
I20260926 06:18:22.004117 18474 ts_tablet_manager.cc:1428] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 06:18:22.005479 18860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:22.000758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51326 (local address 127.2.110.194:41841)
0926 06:18:22.000956 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 06:18:22.000961 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:18:22.001645 (+ 684us) server_negotiation.cc:407] Connection header received
0926 06:18:22.001722 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:18:22.001724 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:18:22.001742 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:18:22.001873 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:18:22.001979 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:22.003072 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:22.004347 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:22.005102 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:22.005168 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:22.005229 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 06:18:22.005352 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 06:18:22.005390 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":92,"threads_started":1}
I20260926 06:18:22.005481 18858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:22.000512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.194:41841 (local address 127.0.0.1:51326)
0926 06:18:22.001429 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:22.001440 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:18:22.001499 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:18:22.001878 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:18:22.001879 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:18:22.001882 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:18:22.001924 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:18:22.001926 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:18:22.003177 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:18:22.003178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:18:22.004255 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:18:22.004260 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:18:22.005167 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:18:22.005168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:18:22.005235 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:22.005256 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 06:18:22.005315 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 06:18:22.005399 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":903,"thread_start_us":84,"threads_started":1}
I20260926 06:18:22.006265 18589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:22.000470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.195:39507 (local address 127.0.0.1:47518)
0926 06:18:22.000521 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:22.000534 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:18:22.000582 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:18:22.001795 (+ 1213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:18:22.001797 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:18:22.001799 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:18:22.001855 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:18:22.001857 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:18:22.003163 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:18:22.003164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:18:22.005248 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:18:22.005251 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:18:22.006004 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:18:22.006005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:18:22.006079 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:22.006085 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 06:18:22.006145 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 06:18:22.006193 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":930}
I20260926 06:18:22.006283 18859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:22.000563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47518 (local address 127.2.110.195:39507)
0926 06:18:22.001618 (+ 1055us) server_negotiation.cc:206] Beginning negotiation
0926 06:18:22.001623 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:18:22.001633 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:18:22.001672 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:18:22.001674 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:18:22.001690 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:18:22.001771 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:18:22.001942 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:22.003062 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:22.005375 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:22.005898 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:22.006003 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:22.006069 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 06:18:22.006158 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 06:18:22.006200 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1023,"thread_start_us":105,"threads_started":1}
I20260926 06:18:22.006611 18654 tablet_service.cc:1467] Processing CreateTablet for tablet 5ec03b6813aa499d8e035917ba6ea022 (DEFAULT_TABLE table=test-table [id=9b6bb26605b64d2aaa71b7884ff41472]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:18:22.006857 18786 tablet_service.cc:1467] Processing CreateTablet for tablet 5ec03b6813aa499d8e035917ba6ea022 (DEFAULT_TABLE table=test-table [id=9b6bb26605b64d2aaa71b7884ff41472]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:18:22.006991 18654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ec03b6813aa499d8e035917ba6ea022. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:18:22.007143 18786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ec03b6813aa499d8e035917ba6ea022. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:18:22.010057 18862 tablet_bootstrap.cc:492] T 5ec03b6813aa499d8e035917ba6ea022 P 9ef9321e79e349e4af20cc546ef9e333: Bootstrap starting.
I20260926 06:18:22.010311 18863 tablet_bootstrap.cc:492] T 5ec03b6813aa499d8e035917ba6ea022 P 66143d40989641d3a0da9f5e895b22b4: Bootstrap starting.
I20260926 06:18:22.010896 18862 tablet_bootstrap.cc:654] T 5ec03b6813aa499d8e035917ba6ea022 P 9ef9321e79e349e4af20cc546ef9e333: Neither blocks nor log segments found. Creating new log.
I20260926 06:18:22.010982 18863 tablet_bootstrap.cc:654] T 5ec03b6813aa499d8e035917ba6ea022 P 66143d40989641d3a0da9f5e895b22b4: Neither blocks nor log segments found. Creating new log.
I20260926 06:18:22.011588 18862 log.cc:824] T 5ec03b6813aa499d8e035917ba6ea022 P 9ef9321e79e349e4af20cc546ef9e333: Log is configured to *not* fsync() on all Append() calls
I20260926 06:18:22.011600 18863 log.cc:824] T 5ec03b6813aa499d8e035917ba6ea022 P 66143d40989641d3a0da9f5e895b22b4: Log is configured to *not* fsync() on all Append() calls
I20260926 06:18:22.012364 18863 tablet_bootstrap.cc:492] T 5ec03b6813aa499d8e035917ba6ea022 P 66143d40989641d3a0da9f5e895b22b4: No bootstrap required, opened a new log
I20260926 06:18:22.012441 18863 ts_tablet_manager.cc:1397] T 5ec03b6813aa499d8e035917ba6ea022 P 66143d40989641d3a0da9f5e895b22b4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:18:22.012516 18862 tablet_bootstrap.cc:492] T 5ec03b6813aa499d8e035917ba6ea022 P 9ef9321e79e349e4af20cc546ef9e333: No bootstrap required, opened a new log
I20260926 06:18:22.012584 18862 ts_tablet_manager.cc:1397] T 5ec03b6813aa499d8e035917ba6ea022 P 9ef9321e79e349e4af20cc546ef9e333: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 06:18:22.014638 18862 raft_consensus.cc:348] T 5ec03b6813aa499d8e035917ba6ea022 P 9ef9321e79e349e4af20cc546ef9e333 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef9321e79e349e4af20cc546ef9e333" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 39507 } } peers { permanent_uuid: "66143d40989641d3a0da9f5e895b22b4" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 41841 } } peers { permanent_uuid: "e559cfe3ebf84f7bb26952adfa617feb" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 41177 } }
I20260926 06:18:22.014638 18863 raft_consensus.cc:348] T 5ec03b6813aa499d8e035917ba6ea022 P 66143d40989641d3a0da9f5e895b22b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef9321e79e349e4af20cc546ef9e333" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 39507 } } peers { permanent_uuid: "66143d40989641d3a0da9f5e895b22b4" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 41841 } } peers { permanent_uuid: "e559cfe3ebf84f7bb26952adfa617feb" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 41177 } }
I20260926 06:18:22.014844 18863 raft_consensus.cc:374] T 5ec03b6813aa499d8e035917ba6ea022 P 66143d40989641d3a0da9f5e895b22b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:18:22.014844 18862 raft_consensus.cc:374] T 5ec03b6813aa499d8e035917ba6ea022 P 9ef9321e79e349e4af20cc546ef9e333 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:18:22.014882 18863 raft_consensus.cc:729] T 5ec03b6813aa499d8e035917ba6ea022 P 66143d40989641d3a0da9f5e895b22b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66143d40989641d3a0da9f5e895b22b4, State: Initialized, Role: FOLLOWER
I20260926 06:18:22.014882 18862 raft_consensus.cc:729] T 5ec03b6813aa499d8e035917ba6ea022 P 9ef9321e79e349e4af20cc546ef9e333 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef9321e79e349e4af20cc546ef9e333, State: Initialized, Role: FOLLOWER
I20260926 06:18:22.015007 18862 consensus_queue.cc:260] T 5ec03b6813aa499d8e035917ba6ea022 P 9ef9321e79e349e4af20cc546ef9e333 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef9321e79e349e4af20cc546ef9e333" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 39507 } } peers { permanent_uuid: "66143d40989641d3a0da9f5e895b22b4" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 41841 } } peers { permanent_uuid: "e559cfe3ebf84f7bb26952adfa617feb" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 41177 } }
I20260926 06:18:22.015007 18863 consensus_queue.cc:260] T 5ec03b6813aa499d8e035917ba6ea022 P 66143d40989641d3a0da9f5e895b22b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef9321e79e349e4af20cc546ef9e333" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 39507 } } peers { permanent_uuid: "66143d40989641d3a0da9f5e895b22b4" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 41841 } } peers { permanent_uuid: "e559cfe3ebf84f7bb26952adfa617feb" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 41177 } }
I20260926 06:18:22.015322 18862 ts_tablet_manager.cc:1428] T 5ec03b6813aa499d8e035917ba6ea022 P 9ef9321e79e349e4af20cc546ef9e333: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 06:18:22.015370 18863 ts_tablet_manager.cc:1428] T 5ec03b6813aa499d8e035917ba6ea022 P 66143d40989641d3a0da9f5e895b22b4: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 06:18:22.015336 18852 heartbeater.cc:503] Master 127.2.110.254:42699 was elected leader, sending a full tablet report...
I20260926 06:18:22.015466 18720 heartbeater.cc:503] Master 127.2.110.254:42699 was elected leader, sending a full tablet report...
W20260926 06:18:22.081074 18721 tablet.cc:2369] T 5ec03b6813aa499d8e035917ba6ea022 P 66143d40989641d3a0da9f5e895b22b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 06:18:22.190011 18583 tablet.cc:2369] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 06:18:22.222725 18853 tablet.cc:2369] T 5ec03b6813aa499d8e035917ba6ea022 P 9ef9321e79e349e4af20cc546ef9e333: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:18:22.323203 18588 raft_consensus.cc:482] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:18:22.323498 18588 raft_consensus.cc:504] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef9321e79e349e4af20cc546ef9e333" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 39507 } } peers { permanent_uuid: "66143d40989641d3a0da9f5e895b22b4" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 41841 } } peers { permanent_uuid: "e559cfe3ebf84f7bb26952adfa617feb" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 41177 } }
I20260926 06:18:22.323956 18588 leader_election.cc:288] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ef9321e79e349e4af20cc546ef9e333 (127.2.110.195:39507), 66143d40989641d3a0da9f5e895b22b4 (127.2.110.194:41841)
I20260926 06:18:22.327250 18868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:22.324162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.195:39507 (local address 127.2.110.193:45405)
0926 06:18:22.324346 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:22.324359 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:18:22.324466 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:18:22.324664 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:18:22.324665 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:18:22.324668 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:18:22.324737 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:18:22.324744 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:18:22.325520 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:18:22.325521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:18:22.326364 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:18:22.326366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:18:22.326957 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:18:22.326958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:18:22.327030 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:22.327047 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 06:18:22.327118 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 06:18:22.327168 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":67,"threads_started":1}
I20260926 06:18:22.327250 18859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:22.324277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:45405 (local address 127.2.110.195:39507)
0926 06:18:22.324323 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 06:18:22.324326 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:18:22.324443 (+ 117us) server_negotiation.cc:407] Connection header received
0926 06:18:22.324562 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:18:22.324563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:18:22.324572 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:18:22.324645 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:18:22.324854 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:22.325425 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:22.326469 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:22.326867 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:22.326957 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:22.326968 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 06:18:22.327125 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 06:18:22.327168 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 06:18:22.327672 18869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:22.324214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.194:41841 (local address 127.2.110.193:53777)
0926 06:18:22.324372 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:22.324382 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:18:22.324592 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:18:22.324870 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:18:22.324872 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:18:22.324875 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:18:22.324992 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:18:22.324994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:18:22.325741 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:18:22.325742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:18:22.326675 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:18:22.326678 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:18:22.327424 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:18:22.327426 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:18:22.327502 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:22.327506 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:18:22.327561 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 06:18:22.327607 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":90,"threads_started":1}
I20260926 06:18:22.327680 18860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:22.324274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:53777 (local address 127.2.110.194:41841)
0926 06:18:22.324323 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 06:18:22.324326 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:18:22.324557 (+ 231us) server_negotiation.cc:407] Connection header received
0926 06:18:22.324748 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:18:22.324750 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:18:22.324763 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:18:22.324862 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:18:22.325071 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:22.325630 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:22.326816 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:22.327237 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:22.327338 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:22.327347 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 06:18:22.327570 (+ 223us) server_negotiation.cc:299] Negotiation successful
0926 06:18:22.327617 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 06:18:22.327651 18806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ec03b6813aa499d8e035917ba6ea022" candidate_uuid: "e559cfe3ebf84f7bb26952adfa617feb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef9321e79e349e4af20cc546ef9e333" is_pre_election: true
I20260926 06:18:22.327860 18806 raft_consensus.cc:2454] T 5ec03b6813aa499d8e035917ba6ea022 P 9ef9321e79e349e4af20cc546ef9e333 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e559cfe3ebf84f7bb26952adfa617feb in term 0.
I20260926 06:18:22.328115 18469 leader_election.cc:302] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ef9321e79e349e4af20cc546ef9e333, e559cfe3ebf84f7bb26952adfa617feb; no voters:
I20260926 06:18:22.328187 18674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ec03b6813aa499d8e035917ba6ea022" candidate_uuid: "e559cfe3ebf84f7bb26952adfa617feb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66143d40989641d3a0da9f5e895b22b4" is_pre_election: true
I20260926 06:18:22.328272 18588 raft_consensus.cc:2781] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:18:22.328336 18588 raft_consensus.cc:482] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:18:22.328349 18674 raft_consensus.cc:2454] T 5ec03b6813aa499d8e035917ba6ea022 P 66143d40989641d3a0da9f5e895b22b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e559cfe3ebf84f7bb26952adfa617feb in term 0.
I20260926 06:18:22.328370 18588 raft_consensus.cc:3037] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:18:22.329068 18588 raft_consensus.cc:504] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef9321e79e349e4af20cc546ef9e333" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 39507 } } peers { permanent_uuid: "66143d40989641d3a0da9f5e895b22b4" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 41841 } } peers { permanent_uuid: "e559cfe3ebf84f7bb26952adfa617feb" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 41177 } }
I20260926 06:18:22.329228 18588 leader_election.cc:288] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb [CANDIDATE]: Term 1 election: Requested vote from peers 9ef9321e79e349e4af20cc546ef9e333 (127.2.110.195:39507), 66143d40989641d3a0da9f5e895b22b4 (127.2.110.194:41841)
I20260926 06:18:22.329381 18806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ec03b6813aa499d8e035917ba6ea022" candidate_uuid: "e559cfe3ebf84f7bb26952adfa617feb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef9321e79e349e4af20cc546ef9e333"
I20260926 06:18:22.329408 18674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ec03b6813aa499d8e035917ba6ea022" candidate_uuid: "e559cfe3ebf84f7bb26952adfa617feb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66143d40989641d3a0da9f5e895b22b4"
I20260926 06:18:22.329481 18674 raft_consensus.cc:3037] T 5ec03b6813aa499d8e035917ba6ea022 P 66143d40989641d3a0da9f5e895b22b4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:18:22.329487 18806 raft_consensus.cc:3037] T 5ec03b6813aa499d8e035917ba6ea022 P 9ef9321e79e349e4af20cc546ef9e333 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:18:22.330264 18806 raft_consensus.cc:2454] T 5ec03b6813aa499d8e035917ba6ea022 P 9ef9321e79e349e4af20cc546ef9e333 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e559cfe3ebf84f7bb26952adfa617feb in term 1.
I20260926 06:18:22.330313 18674 raft_consensus.cc:2454] T 5ec03b6813aa499d8e035917ba6ea022 P 66143d40989641d3a0da9f5e895b22b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e559cfe3ebf84f7bb26952adfa617feb in term 1.
I20260926 06:18:22.330621 18469 leader_election.cc:302] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ef9321e79e349e4af20cc546ef9e333, e559cfe3ebf84f7bb26952adfa617feb; no voters:
I20260926 06:18:22.330807 18588 raft_consensus.cc:2781] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:18:22.330989 18588 raft_consensus.cc:686] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb [term 1 LEADER]: Becoming Leader. State: Replica: e559cfe3ebf84f7bb26952adfa617feb, State: Running, Role: LEADER
I20260926 06:18:22.331090 18588 consensus_queue.cc:237] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef9321e79e349e4af20cc546ef9e333" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 39507 } } peers { permanent_uuid: "66143d40989641d3a0da9f5e895b22b4" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 41841 } } peers { permanent_uuid: "e559cfe3ebf84f7bb26952adfa617feb" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 41177 } }
I20260926 06:18:22.332104 18392 catalog_manager.cc:5495] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb reported cstate change: term changed from 0 to 1, leader changed from <none> to e559cfe3ebf84f7bb26952adfa617feb (127.2.110.193). New cstate: current_term: 1 leader_uuid: "e559cfe3ebf84f7bb26952adfa617feb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef9321e79e349e4af20cc546ef9e333" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 39507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66143d40989641d3a0da9f5e895b22b4" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 41841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e559cfe3ebf84f7bb26952adfa617feb" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 41177 } health_report { overall_health: HEALTHY } } }
I20260926 06:18:22.364725 18873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:22.361891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59168 (local address 127.2.110.193:41177)
0926 06:18:22.362072 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 06:18:22.362074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:18:22.362080 (+ 6us) server_negotiation.cc:407] Connection header received
0926 06:18:22.362130 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:18:22.362131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:18:22.362156 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:18:22.362242 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:18:22.362364 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:22.362888 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:22.363691 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:22.363926 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:22.364020 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:22.364228 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:18:22.364289 (+ 61us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 06:18:22.364455 (+ 166us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59168: token expired
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":90,"threads_started":1}
W20260926 06:18:22.364861 18873 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59168: token expired
I20260926 06:18:22.372526 18585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:22.365351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43136 (local address 127.2.110.254:42699)
0926 06:18:22.365432 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 06:18:22.365545 (+ 113us) server_negotiation.cc:399] Waiting for connection header
0926 06:18:22.365565 (+ 20us) server_negotiation.cc:407] Connection header received
0926 06:18:22.365658 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:18:22.365659 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:18:22.365673 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:18:22.365727 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:18:22.366335 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:22.366968 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:22.367829 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:22.368184 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:22.368302 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:22.369747 (+ 1445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:18:22.369748 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:18:22.369749 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:18:22.369757 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:18:22.371299 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:18:22.371671 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:18:22.371672 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:18:22.371672 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:18:22.371731 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:18:22.371942 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:18:22.371943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:18:22.371944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:18:22.372071 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:18:22.372177 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 06:18:22.372321 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 06:18:22.372411 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 06:18:22.376304 18873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:22.373813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59180 (local address 127.2.110.193:41177)
0926 06:18:22.373859 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 06:18:22.373861 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:18:22.373869 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:18:22.373901 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:18:22.373902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:18:22.373911 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:18:22.373960 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:18:22.374050 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:22.374638 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:22.375617 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:22.375822 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:22.375890 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:22.376014 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:18:22.376119 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:18:22.376160 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 06:18:22.376212 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 06:18:22.376232 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 06:18:22.378417 18674 raft_consensus.cc:1261] T 5ec03b6813aa499d8e035917ba6ea022 P 66143d40989641d3a0da9f5e895b22b4 [term 1 FOLLOWER]: Refusing update from remote peer e559cfe3ebf84f7bb26952adfa617feb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:18:22.378785 18806 raft_consensus.cc:1261] T 5ec03b6813aa499d8e035917ba6ea022 P 9ef9321e79e349e4af20cc546ef9e333 [term 1 FOLLOWER]: Refusing update from remote peer e559cfe3ebf84f7bb26952adfa617feb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:18:22.378959 18588 consensus_queue.cc:1035] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb [LEADER]: Connected to new peer: Peer: permanent_uuid: "66143d40989641d3a0da9f5e895b22b4" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 41841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:18:22.379109 18870 consensus_queue.cc:1035] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ef9321e79e349e4af20cc546ef9e333" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 39507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:18:22.381386 18874 mvcc.cc:205] Tried to move back new op lower bound from 7333492745738973184 to 7333492745549021184. Current Snapshot: MvccSnapshot[applied={T|T < 7333492745738973184}]
I20260926 06:18:22.381610 18875 mvcc.cc:205] Tried to move back new op lower bound from 7333492745738973184 to 7333492745549021184. Current Snapshot: MvccSnapshot[applied={T|T < 7333492745738973184}]
I20260926 06:18:22.382131 18876 mvcc.cc:205] Tried to move back new op lower bound from 7333492745738973184 to 7333492745549021184. Current Snapshot: MvccSnapshot[applied={T|T < 7333492745738973184}]
I20260926 06:18:22.384490 18393 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:18:22.384618 18393 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:18:22.385499 18393 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc: Sending DeleteTablet for 3 replicas of tablet 5ec03b6813aa499d8e035917ba6ea022
I20260926 06:18:22.385900 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 18431
I20260926 06:18:22.385883 18654 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ec03b6813aa499d8e035917ba6ea022 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:18:22 UTC) from {username='kudu', principal='kudu/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:51326
I20260926 06:18:22.385938 18786 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ec03b6813aa499d8e035917ba6ea022 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:18:22 UTC) from {username='kudu', principal='kudu/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:47518
I20260926 06:18:22.386771 18884 tablet_replica.cc:330] T 5ec03b6813aa499d8e035917ba6ea022 P 9ef9321e79e349e4af20cc546ef9e333: stopping tablet replica
I20260926 06:18:22.386873 18883 tablet_replica.cc:330] T 5ec03b6813aa499d8e035917ba6ea022 P 66143d40989641d3a0da9f5e895b22b4: stopping tablet replica
I20260926 06:18:22.386896 18884 raft_consensus.cc:2229] T 5ec03b6813aa499d8e035917ba6ea022 P 9ef9321e79e349e4af20cc546ef9e333 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:18:22.387015 18884 raft_consensus.cc:2258] T 5ec03b6813aa499d8e035917ba6ea022 P 9ef9321e79e349e4af20cc546ef9e333 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:18:22.387041 18883 raft_consensus.cc:2229] T 5ec03b6813aa499d8e035917ba6ea022 P 66143d40989641d3a0da9f5e895b22b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:18:22.387223 18883 raft_consensus.cc:2258] T 5ec03b6813aa499d8e035917ba6ea022 P 66143d40989641d3a0da9f5e895b22b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:18:22.387542 18884 ts_tablet_manager.cc:1905] T 5ec03b6813aa499d8e035917ba6ea022 P 9ef9321e79e349e4af20cc546ef9e333: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:18:22.387749 18883 ts_tablet_manager.cc:1905] T 5ec03b6813aa499d8e035917ba6ea022 P 66143d40989641d3a0da9f5e895b22b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:18:22.389318 18884 ts_tablet_manager.cc:1918] T 5ec03b6813aa499d8e035917ba6ea022 P 9ef9321e79e349e4af20cc546ef9e333: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:18:22.389437 18884 log.cc:1196] T 5ec03b6813aa499d8e035917ba6ea022 P 9ef9321e79e349e4af20cc546ef9e333: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-2/wal/wals/5ec03b6813aa499d8e035917ba6ea022
I20260926 06:18:22.389632 18883 ts_tablet_manager.cc:1918] T 5ec03b6813aa499d8e035917ba6ea022 P 66143d40989641d3a0da9f5e895b22b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:18:22.389760 18884 ts_tablet_manager.cc:1939] T 5ec03b6813aa499d8e035917ba6ea022 P 9ef9321e79e349e4af20cc546ef9e333: Deleting consensus metadata
I20260926 06:18:22.389772 18883 log.cc:1196] T 5ec03b6813aa499d8e035917ba6ea022 P 66143d40989641d3a0da9f5e895b22b4: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-1/wal/wals/5ec03b6813aa499d8e035917ba6ea022
I20260926 06:18:22.390165 18883 ts_tablet_manager.cc:1939] T 5ec03b6813aa499d8e035917ba6ea022 P 66143d40989641d3a0da9f5e895b22b4: Deleting consensus metadata
I20260926 06:18:22.390313 18378 catalog_manager.cc:4841] TS 9ef9321e79e349e4af20cc546ef9e333 (127.2.110.195:39507): tablet 5ec03b6813aa499d8e035917ba6ea022 (table test-table [id=9b6bb26605b64d2aaa71b7884ff41472]) successfully deleted
I20260926 06:18:22.390592 18377 catalog_manager.cc:4841] TS 66143d40989641d3a0da9f5e895b22b4 (127.2.110.194:41841): tablet 5ec03b6813aa499d8e035917ba6ea022 (table test-table [id=9b6bb26605b64d2aaa71b7884ff41472]) successfully deleted
W20260926 06:18:22.392580 18380 connection.cc:535] client connection to 127.2.110.193:41177 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 06:18:22.392686 18380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 06:18:22.393200 18589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:22.393040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.193:41177 (local address 127.0.0.1:59196)
0926 06:18:22.393115 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:22.393157 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.193:41177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 06:18:22.393325 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 18587
W20260926 06:18:22.393477 18380 catalog_manager.cc:4584] TS e559cfe3ebf84f7bb26952adfa617feb (127.2.110.193:41177): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5ec03b6813aa499d8e035917ba6ea022: Network error: Client connection negotiation failed: client connection to 127.2.110.193:41177: connect: Connection refused (error 111)
I20260926 06:18:22.399318 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 18723
I20260926 06:18:22.404301 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 18360
I20260926 06:18:42.410732 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:42699
--webserver_interface=127.2.110.254
--webserver_port=40105
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:36009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.110.254:42699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:18:42.497701 18886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:18:42.497921 18886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:18:42.497946 18886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:18:42.499172 18886 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 06:18:42.499219 18886 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 06:18:42.499233 18886 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 06:18:42.499244 18886 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:18:42.499258 18886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:18:42.499271 18886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:18:42.499316 18886 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:18:42.499348 18886 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:18:42.500881 18886 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:36009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.110.254:42699
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:42699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.110.254
--webserver_port=40105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:18:42.501185 18886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:18:42.501485 18886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:18:42.506436 18892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:18:42.506469 18891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:18:42.506484 18894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:18:42.506795 18886 server_base.cc:1034] running on GCE node
I20260926 06:18:42.507018 18886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:18:42.507346 18886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:18:42.508490 18886 hybrid_clock.cc:648] HybridClock initialized: now 1790403522508477 us; error 45 us; skew 500 ppm
Sep 26 06:18:42 dist-test-slave-hndd krb5kdc[17812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403522, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:18:42.511204 18886 init.cc:377] Logged in from keytab as kudu/127.2.110.254@KRBTEST.COM (short username kudu)
I20260926 06:18:42.511687 18886 webserver.cc:460] Webserver started at http://127.2.110.254:40105/ using document root <none> and password file <none>
I20260926 06:18:42.511875 18886 fs_manager.cc:359] Metadata directory not provided
I20260926 06:18:42.511916 18886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:18:42.513273 18886 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:18:42.514062 18901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:18:42.514266 18886 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:18:42.514354 18886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "c8235472aa12413abcb030e25ed11bfc"
format_stamp: "Formatted at 2026-09-26 06:18:00 on dist-test-slave-hndd"
server_key: "272bd0b7ba819a640db60dc1f473c9f6"
server_key_iv: "e3ddca0877505d1ddd96d5ad5f7f11c4"
server_key_version: "encryptionkey@0"
I20260926 06:18:42.514698 18886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:18:42.541069 18886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:18:42.543591 18886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:18:42.543717 18886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:18:42.547837 18886 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.254:42699
I20260926 06:18:42.547931 18953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.254:42699 every 8 connection(s)
I20260926 06:18:42.548382 18886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:18:42.550052 18954 sys_catalog.cc:263] Verifying existing consensus state
I20260926 06:18:42.550779 18954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc: Bootstrap starting.
I20260926 06:18:42.556631 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 18886
I20260926 06:18:42.557068 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.193:41177
--local_ip_for_outbound_sockets=127.2.110.193
--tserver_master_addrs=127.2.110.254:42699
--webserver_port=42771
--webserver_interface=127.2.110.193
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab
--principal=kudu/127.2.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:36009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 06:18:42.560153 18954 log.cc:824] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc: Log is configured to *not* fsync() on all Append() calls
I20260926 06:18:42.562624 18954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 06:18:42.562978 18954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc: Bootstrap complete.
I20260926 06:18:42.565284 18954 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8235472aa12413abcb030e25ed11bfc" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 42699 } }
I20260926 06:18:42.565608 18954 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8235472aa12413abcb030e25ed11bfc, State: Initialized, Role: FOLLOWER
I20260926 06:18:42.565794 18954 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8235472aa12413abcb030e25ed11bfc" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 42699 } }
I20260926 06:18:42.565877 18954 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:18:42.565932 18954 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:18:42.565989 18954 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc [term 2 FOLLOWER]: Advancing to term 3
I20260926 06:18:42.567025 18954 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8235472aa12413abcb030e25ed11bfc" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 42699 } }
I20260926 06:18:42.567162 18954 leader_election.cc:302] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8235472aa12413abcb030e25ed11bfc; no voters:
I20260926 06:18:42.567395 18954 leader_election.cc:288] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 06:18:42.567435 18958 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc [term 3 FOLLOWER]: Leader election won for term 3
I20260926 06:18:42.567550 18958 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc [term 3 LEADER]: Becoming Leader. State: Replica: c8235472aa12413abcb030e25ed11bfc, State: Running, Role: LEADER
I20260926 06:18:42.567682 18954 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:18:42.567708 18958 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8235472aa12413abcb030e25ed11bfc" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 42699 } }
I20260926 06:18:42.568059 18960 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8235472aa12413abcb030e25ed11bfc. Latest consensus state: current_term: 3 leader_uuid: "c8235472aa12413abcb030e25ed11bfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8235472aa12413abcb030e25ed11bfc" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 42699 } } }
I20260926 06:18:42.568073 18959 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "c8235472aa12413abcb030e25ed11bfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8235472aa12413abcb030e25ed11bfc" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 42699 } } }
I20260926 06:18:42.568289 18959 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc [sys.catalog]: This master's current role is: LEADER
I20260926 06:18:42.568234 18960 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc [sys.catalog]: This master's current role is: LEADER
I20260926 06:18:42.569006 18967 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:18:42.569952 18967 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:18:42.570086 18967 catalog_manager.cc:1259] Loaded cluster ID: 8987a235b02f488fa2d72089fdb46881
I20260926 06:18:42.570122 18967 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:18:42.570946 18967 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:18:42.571143 18967 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc: Loaded TSK: 0
I20260926 06:18:42.571201 18967 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c8235472aa12413abcb030e25ed11bfc: Loaded TSK: 1
F20260926 06:18:42.571432 18967 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fde7310d9ad google::LogMessage::Fail() at ??:0
@ 0x7fde7311088a google::LogMessage::SendToLog() at ??:0
@ 0x7fde7310d4de google::LogMessage::Flush() at ??:0
@ 0x7fde7310ea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fde7587a104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fde75877d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fde7b0671cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fde7b06748e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fde7b067594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fde7b02dc48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fde7b037ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fde74fdb0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fde74fd0954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fde75e346db start_thread at ??:0
@ 0x7fde71c0b61f clone at ??:0
W20260926 06:18:42.676153 18956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:18:42.676443 18956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:18:42.676489 18956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:18:42.678012 18956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:18:42.678090 18956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:18:42.678153 18956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.193
I20260926 06:18:42.679924 18956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:36009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab
--principal=kudu/127.2.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.193:41177
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.110.193
--webserver_port=42771
--tserver_master_addrs=127.2.110.254:42699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.193
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:18:42.680233 18956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:18:42.680498 18956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:18:42.687454 18995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:18:42.687469 18998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:18:42.687697 18956 server_base.cc:1034] running on GCE node
W20260926 06:18:42.687469 18996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:18:42.687906 18956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:18:42.688218 18956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:18:42.689443 18956 hybrid_clock.cc:648] HybridClock initialized: now 1790403522689417 us; error 36 us; skew 500 ppm
Sep 26 06:18:42 dist-test-slave-hndd krb5kdc[17812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403522, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:18:42.692029 18956 init.cc:377] Logged in from keytab as kudu/127.2.110.193@KRBTEST.COM (short username kudu)
I20260926 06:18:42.692551 18956 webserver.cc:460] Webserver started at http://127.2.110.193:42771/ using document root <none> and password file <none>
I20260926 06:18:42.692816 18956 fs_manager.cc:359] Metadata directory not provided
I20260926 06:18:42.692888 18956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:18:42.694397 18956 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:18:42.695210 19005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:18:42.695607 18956 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:18:42.695825 18956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-0/wal
uuid: "e559cfe3ebf84f7bb26952adfa617feb"
format_stamp: "Formatted at 2026-09-26 06:18:00 on dist-test-slave-hndd"
server_key: "eeea97a697644d4849d390d10ada85c6"
server_key_iv: "463f9f6720b364bfe68e018124e93a7f"
server_key_version: "encryptionkey@0"
I20260926 06:18:42.696260 18956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:18:42.711373 18956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:18:42.713769 18956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:18:42.713937 18956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:18:42.714246 18956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:18:42.714747 19012 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 06:18:42.716244 18956 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 06:18:42.716302 18956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 06:18:42.716331 18956 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 06:18:42.717144 18956 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 06:18:42.717190 18956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 06:18:42.717448 19012 tablet_bootstrap.cc:492] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb: Bootstrap starting.
I20260926 06:18:42.725301 18956 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.193:41177
I20260926 06:18:42.725512 19119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.193:41177 every 8 connection(s)
I20260926 06:18:42.725931 18956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
I20260926 06:18:42.727954 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 18956
I20260926 06:18:42.728288 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.194:41841
--local_ip_for_outbound_sockets=127.2.110.194
--tserver_master_addrs=127.2.110.254:42699
--webserver_port=46171
--webserver_interface=127.2.110.194
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab
--principal=kudu/127.2.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:36009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 06:18:42.729297 19012 log.cc:824] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb: Log is configured to *not* fsync() on all Append() calls
I20260926 06:18:42.732146 19012 tablet_bootstrap.cc:492] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 06:18:42.732477 19012 tablet_bootstrap.cc:492] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb: Bootstrap complete.
I20260926 06:18:42.733786 19012 ts_tablet_manager.cc:1397] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.008s
I20260926 06:18:42.735551 19012 raft_consensus.cc:348] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef9321e79e349e4af20cc546ef9e333" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 39507 } } peers { permanent_uuid: "66143d40989641d3a0da9f5e895b22b4" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 41841 } } peers { permanent_uuid: "e559cfe3ebf84f7bb26952adfa617feb" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 41177 } }
I20260926 06:18:42.735917 19012 raft_consensus.cc:729] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e559cfe3ebf84f7bb26952adfa617feb, State: Initialized, Role: FOLLOWER
I20260926 06:18:42.736061 19012 consensus_queue.cc:260] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef9321e79e349e4af20cc546ef9e333" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 39507 } } peers { permanent_uuid: "66143d40989641d3a0da9f5e895b22b4" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 41841 } } peers { permanent_uuid: "e559cfe3ebf84f7bb26952adfa617feb" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 41177 } }
I20260926 06:18:42.736372 19012 ts_tablet_manager.cc:1428] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
W20260926 06:18:42.796422 19122 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:18:42.726710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.193:37739)
0926 06:18:42.727016 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:42.727062 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 06:18:42.727695 (+ 633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:18:42.796191 (+ 68496us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:42699: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":104,"threads_started":1}
W20260926 06:18:42.796689 19120 heartbeater.cc:644] Failed to heartbeat to 127.2.110.254:42699 (0 consecutive failures): Network error: Failed to ping master at 127.2.110.254:42699: Client connection negotiation failed: client connection to 127.2.110.254:42699: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 06:18:42.797026 19122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:42.796907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.193:47899)
0926 06:18:42.796967 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:42.797008 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 06:18:42.797380 19122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:42.797246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.193:56305)
0926 06:18:42.797292 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:42.797315 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 06:18:42.797508 19120 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 06:18:42.818599 19124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:18:42.818895 19124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:18:42.818946 19124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:18:42.820550 19124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:18:42.820653 19124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:18:42.820710 19124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.194
I20260926 06:18:42.822618 19124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:36009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab
--principal=kudu/127.2.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.194:41841
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.110.194
--webserver_port=46171
--tserver_master_addrs=127.2.110.254:42699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.194
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:18:42.822932 19124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:18:42.823218 19124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:18:42.828778 19133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:18:42.828778 19130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:18:42.828789 19131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:18:42.829144 19124 server_base.cc:1034] running on GCE node
I20260926 06:18:42.829685 19124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:18:42.829985 19124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:18:42.831173 19124 hybrid_clock.cc:648] HybridClock initialized: now 1790403522831151 us; error 42 us; skew 500 ppm
Sep 26 06:18:42 dist-test-slave-hndd krb5kdc[17812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403522, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:18:42.833909 19124 init.cc:377] Logged in from keytab as kudu/127.2.110.194@KRBTEST.COM (short username kudu)
I20260926 06:18:42.834370 19124 webserver.cc:460] Webserver started at http://127.2.110.194:46171/ using document root <none> and password file <none>
I20260926 06:18:42.834616 19124 fs_manager.cc:359] Metadata directory not provided
I20260926 06:18:42.834688 19124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:18:42.836218 19124 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:18:42.836946 19140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:18:42.837164 19124 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:18:42.837275 19124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-1/wal
uuid: "66143d40989641d3a0da9f5e895b22b4"
format_stamp: "Formatted at 2026-09-26 06:18:00 on dist-test-slave-hndd"
server_key: "ba5867f04f8dca91116f12cbf7de80f8"
server_key_iv: "bbcc74d120502733fe96a7a1eff028d3"
server_key_version: "encryptionkey@0"
I20260926 06:18:42.837729 19124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:18:42.850759 19124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:18:42.853236 19124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:18:42.853461 19124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:18:42.853747 19124 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:18:42.854075 19124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:18:42.854130 19124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:18:42.854171 19124 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:18:42.854214 19124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:18:42.860355 19124 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.194:41841
I20260926 06:18:42.860400 19253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.194:41841 every 8 connection(s)
I20260926 06:18:42.860756 19124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
I20260926 06:18:42.861867 19256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:42.861493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.194:36435)
0926 06:18:42.861644 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:42.861723 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":67,"threads_started":1}
W20260926 06:18:42.862339 19254 heartbeater.cc:644] Failed to heartbeat to 127.2.110.254:42699 (0 consecutive failures): Network error: Failed to ping master at 127.2.110.254:42699: Client connection negotiation failed: client connection to 127.2.110.254:42699: connect: Connection refused (error 111)
I20260926 06:18:42.862923 19256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:42.862696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.194:43441)
0926 06:18:42.862860 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:42.862900 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
I20260926 06:18:42.863543 19256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:42.863329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.194:46397)
0926 06:18:42.863444 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:42.863501 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260926 06:18:42.863817 19254 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 06:18:42.864482 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 19124
I20260926 06:18:42.864809 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.195:39507
--local_ip_for_outbound_sockets=127.2.110.195
--tserver_master_addrs=127.2.110.254:42699
--webserver_port=40435
--webserver_interface=127.2.110.195
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab
--principal=kudu/127.2.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:36009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:18:42.947381 19257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:18:42.947660 19257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:18:42.947703 19257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:18:42.949122 19257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:18:42.949231 19257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:18:42.949285 19257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.195
I20260926 06:18:42.950941 19257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:36009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab
--principal=kudu/127.2.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.195:39507
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.110.195
--webserver_port=40435
--tserver_master_addrs=127.2.110.254:42699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.195
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:18:42.951229 19257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:18:42.951493 19257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:18:42.956909 19263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:18:42.956962 19265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:18:42.957098 19257 server_base.cc:1034] running on GCE node
W20260926 06:18:42.956861 19262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:18:42.957391 19257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:18:42.957662 19257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:18:42.958842 19257 hybrid_clock.cc:648] HybridClock initialized: now 1790403522958818 us; error 36 us; skew 500 ppm
Sep 26 06:18:42 dist-test-slave-hndd krb5kdc[17812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403522, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:18:42.961854 19257 init.cc:377] Logged in from keytab as kudu/127.2.110.195@KRBTEST.COM (short username kudu)
I20260926 06:18:42.962261 19257 webserver.cc:460] Webserver started at http://127.2.110.195:40435/ using document root <none> and password file <none>
I20260926 06:18:42.962466 19257 fs_manager.cc:359] Metadata directory not provided
I20260926 06:18:42.962507 19257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:18:42.963903 19257 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:18:42.964821 19272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:18:42.965061 19257 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:18:42.965169 19257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-2/wal
uuid: "9ef9321e79e349e4af20cc546ef9e333"
format_stamp: "Formatted at 2026-09-26 06:18:01 on dist-test-slave-hndd"
server_key: "3975f58e155ce13dc1511482e4208a6b"
server_key_iv: "513b5d4d4999168638eefb54169f83e8"
server_key_version: "encryptionkey@0"
I20260926 06:18:42.965616 19257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:18:42.978706 19257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:18:42.981045 19257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:18:42.981166 19257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:18:42.981427 19257 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:18:42.981788 19257 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:18:42.981823 19257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:18:42.981848 19257 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:18:42.981889 19257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:18:42.988538 19257 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.195:39507
I20260926 06:18:42.988582 19385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.195:39507 every 8 connection(s)
I20260926 06:18:42.988974 19257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
I20260926 06:18:42.990257 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 19257
I20260926 06:18:42.990356 19388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:42.989820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.195:44333)
0926 06:18:42.989990 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:42.990139 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":74,"threads_started":1}
W20260926 06:18:42.990803 19386 heartbeater.cc:644] Failed to heartbeat to 127.2.110.254:42699 (0 consecutive failures): Network error: Failed to ping master at 127.2.110.254:42699: Client connection negotiation failed: client connection to 127.2.110.254:42699: connect: Connection refused (error 111)
I20260926 06:18:42.991137 19388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:42.991022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.195:47357)
0926 06:18:42.991084 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:42.991115 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 06:18:42.991532 19388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:42.991415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.195:39095)
0926 06:18:42.991477 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:42.991510 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 06:18:42.991742 19386 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 06:18:43.798370 19390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:43.798034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.193:54353)
0926 06:18:43.798233 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:43.798287 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":96,"threads_started":1}
I20260926 06:18:43.864513 19391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:43.864246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.194:52997)
0926 06:18:43.864409 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:43.864452 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":63,"threads_started":1}
I20260926 06:18:43.992705 19392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:43.992390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.195:54939)
0926 06:18:43.992594 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:43.992645 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":99,"threads_started":1}
I20260926 06:18:44.377930 19393 raft_consensus.cc:482] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:18:44.378259 19393 raft_consensus.cc:504] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef9321e79e349e4af20cc546ef9e333" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 39507 } } peers { permanent_uuid: "66143d40989641d3a0da9f5e895b22b4" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 41841 } } peers { permanent_uuid: "e559cfe3ebf84f7bb26952adfa617feb" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 41177 } }
I20260926 06:18:44.378669 19393 leader_election.cc:288] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9ef9321e79e349e4af20cc546ef9e333 (127.2.110.195:39507), 66143d40989641d3a0da9f5e895b22b4 (127.2.110.194:41841)
Sep 26 06:18:44 dist-test-slave-hndd krb5kdc[17812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403522, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.193@KRBTEST.COM for kudu/127.2.110.194@KRBTEST.COM
Sep 26 06:18:44 dist-test-slave-hndd krb5kdc[17812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403522, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.193@KRBTEST.COM for kudu/127.2.110.195@KRBTEST.COM
I20260926 06:18:44.397151 19394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:44.378889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.195:39507 (local address 127.2.110.193:53259)
0926 06:18:44.379201 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:44.379215 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 06:18:44.379316 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:18:44.380444 (+ 1128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:18:44.380447 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:18:44.381172 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:18:44.381488 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:18:44.381498 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:18:44.383473 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:18:44.383474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:18:44.384329 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:18:44.384331 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:18:44.385016 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:18:44.385017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:18:44.385127 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:44.391073 (+ 5946us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:18:44.391079 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:18:44.393188 (+ 2109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:18:44.395819 (+ 2631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:18:44.395820 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:18:44.395820 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:18:44.395994 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:18:44.396254 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:18:44.396255 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:18:44.396256 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:18:44.396320 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:18:44.396700 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:18:44.396701 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:18:44.396747 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 06:18:44.396845 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 06:18:44.397056 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":8,"thread_start_us":86,"threads_started":1}
I20260926 06:18:44.397168 19396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:44.379033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:53259 (local address 127.2.110.195:39507)
0926 06:18:44.379243 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 06:18:44.379256 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 06:18:44.379298 (+ 42us) server_negotiation.cc:407] Connection header received
0926 06:18:44.380004 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:18:44.380008 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:18:44.380072 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:18:44.380246 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:18:44.381674 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:44.383312 (+ 1638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:44.384451 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:44.384894 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:44.385039 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:44.393321 (+ 8282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:18:44.393325 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:18:44.393325 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:18:44.393381 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:18:44.395742 (+ 2361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:18:44.396083 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:18:44.396086 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:18:44.396087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:18:44.396180 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:18:44.396387 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:18:44.396388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:18:44.396389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:18:44.396624 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:18:44.396718 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 06:18:44.396903 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 06:18:44.397044 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":70,"threads_started":1}
I20260926 06:18:44.397181 19397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:44.379052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:60827 (local address 127.2.110.194:41841)
0926 06:18:44.379243 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 06:18:44.379256 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 06:18:44.379389 (+ 133us) server_negotiation.cc:407] Connection header received
0926 06:18:44.380004 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:18:44.380008 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:18:44.380072 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:18:44.380246 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:18:44.381674 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:44.383296 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:44.384480 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:44.384887 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:44.385015 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:44.393068 (+ 8053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:18:44.393074 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:18:44.393074 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:18:44.393088 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:18:44.395211 (+ 2123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:18:44.395778 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:18:44.395781 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:18:44.395781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:18:44.395875 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:18:44.396193 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:18:44.396194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:18:44.396194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:18:44.396466 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:18:44.396598 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 06:18:44.396894 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 06:18:44.397038 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":61,"threads_started":1}
I20260926 06:18:44.397605 19395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:44.378889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.194:41841 (local address 127.2.110.193:60827)
0926 06:18:44.379302 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:44.379313 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:18:44.379372 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:18:44.380442 (+ 1070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:18:44.380447 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:18:44.381172 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:18:44.381489 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:18:44.381498 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:18:44.383487 (+ 1989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:18:44.383488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:18:44.384329 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:18:44.384332 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:18:44.385011 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:18:44.385012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:18:44.385126 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:44.391063 (+ 5937us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:18:44.391074 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:18:44.392897 (+ 1823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:18:44.395318 (+ 2421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:18:44.395327 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:18:44.395330 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:18:44.395669 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:18:44.395976 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:18:44.395977 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:18:44.395977 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:18:44.396107 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:18:44.396578 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:18:44.396585 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:18:44.396723 (+ 138us) client_negotiation.cc:769] Sending connection context
0926 06:18:44.396840 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 06:18:44.397144 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"mutex_wait_us":207,"thread_start_us":81,"threads_started":1}
W20260926 06:18:44.398074 19007 leader_election.cc:341] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9ef9321e79e349e4af20cc546ef9e333 (127.2.110.195:39507): Not found: Tablet not found: 5ec03b6813aa499d8e035917ba6ea022
W20260926 06:18:44.398263 19006 leader_election.cc:341] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 66143d40989641d3a0da9f5e895b22b4 (127.2.110.194:41841): Not found: Tablet not found: 5ec03b6813aa499d8e035917ba6ea022
I20260926 06:18:44.398320 19006 leader_election.cc:302] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e559cfe3ebf84f7bb26952adfa617feb; no voters: 66143d40989641d3a0da9f5e895b22b4, 9ef9321e79e349e4af20cc546ef9e333
I20260926 06:18:44.398433 19393 raft_consensus.cc:2726] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 06:18:44.799139 19395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:44.798969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.193:47085)
0926 06:18:44.799039 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:44.799085 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 06:18:44.865458 19398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:44.865071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.194:43553)
0926 06:18:44.865214 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:44.865266 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":66,"threads_started":1}
I20260926 06:18:44.993652 19399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:44.993396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.195:60787)
0926 06:18:44.993557 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:44.993607 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":70,"threads_started":1}
I20260926 06:18:45.800177 19400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:45.799772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.193:50663)
0926 06:18:45.799987 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:45.800054 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":94,"threads_started":1}
I20260926 06:18:45.866307 19401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:45.866074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.194:56337)
0926 06:18:45.866219 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:45.866260 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":98,"threads_started":1}
I20260926 06:18:45.994606 19402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:45.994248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.195:44451)
0926 06:18:45.994457 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:45.994531 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":67,"threads_started":1}
I20260926 06:18:46.533695 19403 raft_consensus.cc:482] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:18:46.533800 19403 raft_consensus.cc:504] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef9321e79e349e4af20cc546ef9e333" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 39507 } } peers { permanent_uuid: "66143d40989641d3a0da9f5e895b22b4" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 41841 } } peers { permanent_uuid: "e559cfe3ebf84f7bb26952adfa617feb" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 41177 } }
I20260926 06:18:46.534088 19403 leader_election.cc:288] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9ef9321e79e349e4af20cc546ef9e333 (127.2.110.195:39507), 66143d40989641d3a0da9f5e895b22b4 (127.2.110.194:41841)
W20260926 06:18:46.534562 19006 leader_election.cc:341] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 66143d40989641d3a0da9f5e895b22b4 (127.2.110.194:41841): Not found: Tablet not found: 5ec03b6813aa499d8e035917ba6ea022
W20260926 06:18:46.534628 19007 leader_election.cc:341] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9ef9321e79e349e4af20cc546ef9e333 (127.2.110.195:39507): Not found: Tablet not found: 5ec03b6813aa499d8e035917ba6ea022
I20260926 06:18:46.534718 19007 leader_election.cc:302] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e559cfe3ebf84f7bb26952adfa617feb; no voters: 66143d40989641d3a0da9f5e895b22b4, 9ef9321e79e349e4af20cc546ef9e333
I20260926 06:18:46.534821 19403 raft_consensus.cc:2726] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 06:18:46.801067 19404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:46.800790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.193:55187)
0926 06:18:46.800951 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:46.801001 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":66,"threads_started":1}
I20260926 06:18:46.867600 19405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:46.867084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.194:50457)
0926 06:18:46.867255 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:46.867449 (+ 194us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":82,"threads_started":1}
I20260926 06:18:46.995703 19406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:46.995408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.195:46153)
0926 06:18:46.995580 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:46.995628 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":86,"threads_started":1}
I20260926 06:18:47.801910 19407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:47.801660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.193:37227)
0926 06:18:47.801816 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:47.801862 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":68,"threads_started":1}
I20260926 06:18:47.868582 19408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:47.868220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.194:33611)
0926 06:18:47.868464 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:47.868518 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":88,"threads_started":1}
I20260926 06:18:47.996590 19409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:47.996332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.195:33441)
0926 06:18:47.996495 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:47.996541 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":88,"threads_started":1}
I20260926 06:18:48.802908 19410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:48.802636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.193:52513)
0926 06:18:48.802814 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:48.802860 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":96,"threads_started":1}
I20260926 06:18:48.869499 19411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:48.869185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.194:47985)
0926 06:18:48.869393 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:48.869445 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":111,"threads_started":1}
I20260926 06:18:48.997639 19412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:48.997282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.195:58813)
0926 06:18:48.997546 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:48.997591 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":162,"threads_started":1}
I20260926 06:18:49.609787 19413 raft_consensus.cc:482] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:18:49.609903 19413 raft_consensus.cc:504] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef9321e79e349e4af20cc546ef9e333" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 39507 } } peers { permanent_uuid: "66143d40989641d3a0da9f5e895b22b4" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 41841 } } peers { permanent_uuid: "e559cfe3ebf84f7bb26952adfa617feb" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 41177 } }
I20260926 06:18:49.610155 19413 leader_election.cc:288] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9ef9321e79e349e4af20cc546ef9e333 (127.2.110.195:39507), 66143d40989641d3a0da9f5e895b22b4 (127.2.110.194:41841)
W20260926 06:18:49.610596 19006 leader_election.cc:341] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 66143d40989641d3a0da9f5e895b22b4 (127.2.110.194:41841): Not found: Tablet not found: 5ec03b6813aa499d8e035917ba6ea022
W20260926 06:18:49.610687 19007 leader_election.cc:341] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9ef9321e79e349e4af20cc546ef9e333 (127.2.110.195:39507): Not found: Tablet not found: 5ec03b6813aa499d8e035917ba6ea022
I20260926 06:18:49.610730 19007 leader_election.cc:302] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e559cfe3ebf84f7bb26952adfa617feb; no voters: 66143d40989641d3a0da9f5e895b22b4, 9ef9321e79e349e4af20cc546ef9e333
I20260926 06:18:49.610836 19413 raft_consensus.cc:2726] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 06:18:49.804064 19414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:49.803736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.193:53335)
0926 06:18:49.803961 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:49.803998 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":151,"threads_started":1}
I20260926 06:18:49.870256 19415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:49.870032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.194:53279)
0926 06:18:49.870164 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:49.870206 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":66,"threads_started":1}
I20260926 06:18:49.998603 19416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:49.998319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.195:44357)
0926 06:18:49.998506 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:49.998551 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":100,"threads_started":1}
I20260926 06:18:50.805055 19417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:50.804754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.193:38239)
0926 06:18:50.804953 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:50.805004 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":72,"threads_started":1}
I20260926 06:18:50.871181 19418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:50.870926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.194:58395)
0926 06:18:50.871085 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:50.871131 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":73,"threads_started":1}
I20260926 06:18:50.999902 19419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:50.999457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.195:40687)
0926 06:18:50.999802 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:50.999851 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":227,"threads_started":1}
I20260926 06:18:51.805996 19420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:51.805741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.193:57219)
0926 06:18:51.805900 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:51.805938 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":68,"threads_started":1}
I20260926 06:18:51.824496 19421 raft_consensus.cc:482] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:18:51.824610 19421 raft_consensus.cc:504] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef9321e79e349e4af20cc546ef9e333" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 39507 } } peers { permanent_uuid: "66143d40989641d3a0da9f5e895b22b4" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 41841 } } peers { permanent_uuid: "e559cfe3ebf84f7bb26952adfa617feb" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 41177 } }
I20260926 06:18:51.824803 19421 leader_election.cc:288] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9ef9321e79e349e4af20cc546ef9e333 (127.2.110.195:39507), 66143d40989641d3a0da9f5e895b22b4 (127.2.110.194:41841)
W20260926 06:18:51.825412 19006 leader_election.cc:341] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 66143d40989641d3a0da9f5e895b22b4 (127.2.110.194:41841): Not found: Tablet not found: 5ec03b6813aa499d8e035917ba6ea022
W20260926 06:18:51.825492 19007 leader_election.cc:341] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9ef9321e79e349e4af20cc546ef9e333 (127.2.110.195:39507): Not found: Tablet not found: 5ec03b6813aa499d8e035917ba6ea022
I20260926 06:18:51.825554 19007 leader_election.cc:302] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e559cfe3ebf84f7bb26952adfa617feb; no voters: 66143d40989641d3a0da9f5e895b22b4, 9ef9321e79e349e4af20cc546ef9e333
I20260926 06:18:51.825655 19421 raft_consensus.cc:2726] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 06:18:51.871994 19422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:51.871765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.194:39445)
0926 06:18:51.871909 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:51.871947 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":85,"threads_started":1}
I20260926 06:18:52.000838 19423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:52.000620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.195:36543)
0926 06:18:52.000746 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:52.000790 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":63,"threads_started":1}
I20260926 06:18:52.806923 19424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:52.806668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.193:48885)
0926 06:18:52.806824 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:52.806873 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":73,"threads_started":1}
I20260926 06:18:52.872709 19425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:52.872499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.194:58689)
0926 06:18:52.872625 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:52.872660 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":70,"threads_started":1}
I20260926 06:18:53.001688 19426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:53.001440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.195:54143)
0926 06:18:53.001581 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:53.001621 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":75,"threads_started":1}
I20260926 06:18:53.807816 19427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:53.807552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.193:46087)
0926 06:18:53.807718 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:53.807767 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":68,"threads_started":1}
I20260926 06:18:53.873585 19428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:53.873300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.194:45699)
0926 06:18:53.873490 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:53.873535 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":91,"threads_started":1}
I20260926 06:18:54.002552 19429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:54.002249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.195:39281)
0926 06:18:54.002456 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:54.002502 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":86,"threads_started":1}
I20260926 06:18:54.808772 19430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:54.808530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.193:44249)
0926 06:18:54.808682 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:54.808725 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":69,"threads_started":1}
I20260926 06:18:54.874358 19431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:54.874145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.194:38277)
0926 06:18:54.874270 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:54.874310 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":62,"threads_started":1}
I20260926 06:18:55.003410 19432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:55.003138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.195:35247)
0926 06:18:55.003314 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:55.003361 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":85,"threads_started":1}
I20260926 06:18:55.809900 19433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:55.809658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.193:42345)
0926 06:18:55.809806 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:55.809854 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":63,"threads_started":1}
I20260926 06:18:55.875164 19434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:55.874913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.194:53083)
0926 06:18:55.875068 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:55.875115 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":72,"threads_started":1}
I20260926 06:18:56.004338 19435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:56.004065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.195:59389)
0926 06:18:56.004239 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:56.004286 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":66,"threads_started":1}
I20260926 06:18:56.810796 19436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:56.810532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.193:46911)
0926 06:18:56.810694 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:56.810744 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":84,"threads_started":1}
I20260926 06:18:56.876114 19437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:56.875867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.194:53879)
0926 06:18:56.876023 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:56.876067 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":61,"threads_started":1}
I20260926 06:18:57.005234 19438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:57.004947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.195:53181)
0926 06:18:57.005117 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:57.005163 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":87,"threads_started":1}
I20260926 06:18:57.811866 19439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:57.811515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.193:53439)
0926 06:18:57.811725 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:57.811799 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":70,"threads_started":1}
I20260926 06:18:57.877329 19440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:57.876962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.194:34083)
0926 06:18:57.877206 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:57.877263 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":99,"threads_started":1}
I20260926 06:18:57.888566 19441 raft_consensus.cc:482] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:18:57.888669 19441 raft_consensus.cc:504] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef9321e79e349e4af20cc546ef9e333" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 39507 } } peers { permanent_uuid: "66143d40989641d3a0da9f5e895b22b4" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 41841 } } peers { permanent_uuid: "e559cfe3ebf84f7bb26952adfa617feb" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 41177 } }
I20260926 06:18:57.888870 19441 leader_election.cc:288] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9ef9321e79e349e4af20cc546ef9e333 (127.2.110.195:39507), 66143d40989641d3a0da9f5e895b22b4 (127.2.110.194:41841)
W20260926 06:18:57.889425 19006 leader_election.cc:341] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 66143d40989641d3a0da9f5e895b22b4 (127.2.110.194:41841): Not found: Tablet not found: 5ec03b6813aa499d8e035917ba6ea022
W20260926 06:18:57.889521 19007 leader_election.cc:341] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9ef9321e79e349e4af20cc546ef9e333 (127.2.110.195:39507): Not found: Tablet not found: 5ec03b6813aa499d8e035917ba6ea022
I20260926 06:18:57.889540 19007 leader_election.cc:302] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e559cfe3ebf84f7bb26952adfa617feb; no voters: 66143d40989641d3a0da9f5e895b22b4, 9ef9321e79e349e4af20cc546ef9e333
I20260926 06:18:57.889614 19441 raft_consensus.cc:2726] T 5ec03b6813aa499d8e035917ba6ea022 P e559cfe3ebf84f7bb26952adfa617feb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 06:18:58.001118 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 18956
I20260926 06:18:58.006371 19442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:58.006101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42699 (local address 127.2.110.195:48621)
0926 06:18:58.006281 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:58.006311 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.254:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":102,"threads_started":1}
I20260926 06:18:58.006520 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 19124
I20260926 06:18:58.011219 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 19257
2026-09-26T06:18:58Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58778 ms)
[----------] 30 tests from SecurityITest (245649 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:18:58 dist-test-slave-hndd krb5kdc[19445](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:18:58 dist-test-slave-hndd krb5kdc[19445](info): set up 2 sockets
Sep 26 06:18:58 dist-test-slave-hndd krb5kdc[19445](info): commencing operation
krb5kdc: starting...
W20260926 06:18:59.097172 2491 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Sep 26 06:18:59 dist-test-slave-hndd krb5kdc[19445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:18:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:18:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
I20260926 06:18:59.225193 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:42379
--webserver_interface=127.2.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:39825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.110.254:42379
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:18:59.307538 19458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:18:59.307803 19458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:18:59.307842 19458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:18:59.309007 19458 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:18:59.309088 19458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:18:59.309113 19458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:18:59.309136 19458 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:18:59.309165 19458 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:18:59.310690 19458 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:39825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.110.254:42379
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:42379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:18:59.310981 19458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:18:59.311215 19458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:18:59.316635 19463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:18:59.316592 19466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:18:59.316655 19458 server_base.cc:1034] running on GCE node
W20260926 06:18:59.316598 19464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:18:59.317241 19458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:18:59.317612 19458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:18:59.318786 19458 hybrid_clock.cc:648] HybridClock initialized: now 1790403539318738 us; error 61 us; skew 500 ppm
Sep 26 06:18:59 dist-test-slave-hndd krb5kdc[19445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403539, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:18:59.321442 19458 init.cc:377] Logged in from keytab as kudu/127.2.110.254@KRBTEST.COM (short username kudu)
I20260926 06:18:59.321931 19458 webserver.cc:460] Webserver started at http://127.2.110.254:33943/ using document root <none> and password file <none>
I20260926 06:18:59.322124 19458 fs_manager.cc:359] Metadata directory not provided
I20260926 06:18:59.322165 19458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:18:59.322258 19458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:18:59.323298 19458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/master-0/data/instance:
uuid: "fa5cb55d4a8b4139a7fdc9775dd6916f"
format_stamp: "Formatted at 2026-09-26 06:18:59 on dist-test-slave-hndd"
server_key: "abe4497b2dba0362d682b20d22fee6f9"
server_key_iv: "143105b89b42de6dd3e519166d553e25"
server_key_version: "encryptionkey@0"
I20260926 06:18:59.323611 19458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/master-0/wal/instance:
uuid: "fa5cb55d4a8b4139a7fdc9775dd6916f"
format_stamp: "Formatted at 2026-09-26 06:18:59 on dist-test-slave-hndd"
server_key: "abe4497b2dba0362d682b20d22fee6f9"
server_key_iv: "143105b89b42de6dd3e519166d553e25"
server_key_version: "encryptionkey@0"
I20260926 06:18:59.324986 19458 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:18:59.325862 19473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:18:59.326028 19458 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 06:18:59.326123 19458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "fa5cb55d4a8b4139a7fdc9775dd6916f"
format_stamp: "Formatted at 2026-09-26 06:18:59 on dist-test-slave-hndd"
server_key: "abe4497b2dba0362d682b20d22fee6f9"
server_key_iv: "143105b89b42de6dd3e519166d553e25"
server_key_version: "encryptionkey@0"
I20260926 06:18:59.326185 19458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:18:59.343741 19458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:18:59.346035 19458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:18:59.346155 19458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:18:59.350302 19458 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.254:42379
I20260926 06:18:59.350732 19458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:18:59.350720 19525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.254:42379 every 8 connection(s)
I20260926 06:18:59.351946 19526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:18:59.354485 19526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa5cb55d4a8b4139a7fdc9775dd6916f: Bootstrap starting.
I20260926 06:18:59.355134 19526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa5cb55d4a8b4139a7fdc9775dd6916f: Neither blocks nor log segments found. Creating new log.
I20260926 06:18:59.355444 19526 log.cc:824] T 00000000000000000000000000000000 P fa5cb55d4a8b4139a7fdc9775dd6916f: Log is configured to *not* fsync() on all Append() calls
I20260926 06:18:59.356307 19526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa5cb55d4a8b4139a7fdc9775dd6916f: No bootstrap required, opened a new log
I20260926 06:18:59.358258 19526 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa5cb55d4a8b4139a7fdc9775dd6916f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa5cb55d4a8b4139a7fdc9775dd6916f" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 42379 } }
I20260926 06:18:59.358481 19526 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa5cb55d4a8b4139a7fdc9775dd6916f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:18:59.358534 19526 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa5cb55d4a8b4139a7fdc9775dd6916f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa5cb55d4a8b4139a7fdc9775dd6916f, State: Initialized, Role: FOLLOWER
I20260926 06:18:59.358703 19526 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa5cb55d4a8b4139a7fdc9775dd6916f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa5cb55d4a8b4139a7fdc9775dd6916f" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 42379 } }
I20260926 06:18:59.358788 19526 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa5cb55d4a8b4139a7fdc9775dd6916f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:18:59.358832 19526 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa5cb55d4a8b4139a7fdc9775dd6916f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:18:59.358882 19526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa5cb55d4a8b4139a7fdc9775dd6916f [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:18:59.359489 19526 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa5cb55d4a8b4139a7fdc9775dd6916f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa5cb55d4a8b4139a7fdc9775dd6916f" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 42379 } }
I20260926 06:18:59.359613 19526 leader_election.cc:302] T 00000000000000000000000000000000 P fa5cb55d4a8b4139a7fdc9775dd6916f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa5cb55d4a8b4139a7fdc9775dd6916f; no voters:
I20260926 06:18:59.359823 19526 leader_election.cc:288] T 00000000000000000000000000000000 P fa5cb55d4a8b4139a7fdc9775dd6916f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:18:59.359885 19529 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa5cb55d4a8b4139a7fdc9775dd6916f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:18:59.360097 19526 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa5cb55d4a8b4139a7fdc9775dd6916f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:18:59.360150 19529 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa5cb55d4a8b4139a7fdc9775dd6916f [term 1 LEADER]: Becoming Leader. State: Replica: fa5cb55d4a8b4139a7fdc9775dd6916f, State: Running, Role: LEADER
I20260926 06:18:59.360363 19529 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa5cb55d4a8b4139a7fdc9775dd6916f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa5cb55d4a8b4139a7fdc9775dd6916f" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 42379 } }
I20260926 06:18:59.360819 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 19458
I20260926 06:18:59.360939 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/master-0/wal/instance
I20260926 06:18:59.361186 2491 external_mini_cluster.cc:1428] Setting key 81ce635107902948fca8982708d4ccd3
I20260926 06:18:59.361488 19530 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa5cb55d4a8b4139a7fdc9775dd6916f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa5cb55d4a8b4139a7fdc9775dd6916f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa5cb55d4a8b4139a7fdc9775dd6916f" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 42379 } } }
I20260926 06:18:59.362004 19530 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa5cb55d4a8b4139a7fdc9775dd6916f [sys.catalog]: This master's current role is: LEADER
I20260926 06:18:59.361428 19531 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa5cb55d4a8b4139a7fdc9775dd6916f [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa5cb55d4a8b4139a7fdc9775dd6916f. Latest consensus state: current_term: 1 leader_uuid: "fa5cb55d4a8b4139a7fdc9775dd6916f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa5cb55d4a8b4139a7fdc9775dd6916f" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 42379 } } }
I20260926 06:18:59.362669 19531 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa5cb55d4a8b4139a7fdc9775dd6916f [sys.catalog]: This master's current role is: LEADER
I20260926 06:18:59.363490 19545 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:18:59.363976 19545 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:18:59.365592 19545 catalog_manager.cc:1347] Generated new cluster ID: 1f8f65008bc0413f8b70731bf15fd0e6
I20260926 06:18:59.365693 19545 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:18:59.369977 19545 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:18:59.370779 19545 catalog_manager.cc:1504] Loading token signing keys...
Sep 26 06:18:59 dist-test-slave-hndd krb5kdc[19445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:18:59.376646 19544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:59.361992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50682 (local address 127.2.110.254:42379)
0926 06:18:59.364561 (+ 2569us) server_negotiation.cc:206] Beginning negotiation
0926 06:18:59.364569 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 06:18:59.364594 (+ 25us) server_negotiation.cc:407] Connection header received
0926 06:18:59.364683 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:18:59.364688 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:18:59.364733 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:18:59.364884 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:18:59.365672 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:59.367068 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:59.368172 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:59.368861 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:59.368977 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:59.372308 (+ 3331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:18:59.372313 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:18:59.372314 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:18:59.372329 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:18:59.374539 (+ 2210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:18:59.374982 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:18:59.374986 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:18:59.374986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:18:59.375096 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:18:59.375302 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:18:59.375303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:18:59.375303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:18:59.375603 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:18:59.376150 (+ 547us) server_negotiation.cc:1035] Waiting for connection context
0926 06:18:59.376223 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 06:18:59.376386 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2527,"thread_start_us":664,"threads_started":1}
I20260926 06:18:59.379889 19545 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa5cb55d4a8b4139a7fdc9775dd6916f: Generated new TSK 0
I20260926 06:18:59.380091 19545 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab.
Entry for principal kudu/127.2.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab.
I20260926 06:18:59.409255 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.193:0
--local_ip_for_outbound_sockets=127.2.110.193
--webserver_interface=127.2.110.193
--webserver_port=0
--tserver_master_addrs=127.2.110.254:42379
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab
--principal=kudu/127.2.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:39825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:18:59.493824 19552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:18:59.494041 19552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:18:59.494067 19552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:18:59.495244 19552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:18:59.495283 19552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:18:59.495311 19552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.193
I20260926 06:18:59.496747 19552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:39825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab
--principal=kudu/127.2.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.110.193
--webserver_port=0
--tserver_master_addrs=127.2.110.254:42379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.193
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:18:59.496951 19552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:18:59.497134 19552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:18:59.503191 19557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:18:59.503191 19560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:18:59.503298 19558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:18:59.503314 19552 server_base.cc:1034] running on GCE node
I20260926 06:18:59.503589 19552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:18:59.503859 19552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:18:59.505048 19552 hybrid_clock.cc:648] HybridClock initialized: now 1790403539505023 us; error 40 us; skew 500 ppm
Sep 26 06:18:59 dist-test-slave-hndd krb5kdc[19445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403539, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:18:59.508085 19552 init.cc:377] Logged in from keytab as kudu/127.2.110.193@KRBTEST.COM (short username kudu)
I20260926 06:18:59.508594 19552 webserver.cc:460] Webserver started at http://127.2.110.193:40009/ using document root <none> and password file <none>
I20260926 06:18:59.508852 19552 fs_manager.cc:359] Metadata directory not provided
I20260926 06:18:59.508899 19552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:18:59.509024 19552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:18:59.510527 19552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/ts-0/data/instance:
uuid: "91cd62e43450456aadbf052b9e23c65c"
format_stamp: "Formatted at 2026-09-26 06:18:59 on dist-test-slave-hndd"
server_key: "539f9243aba64468be4925140c29f1fd"
server_key_iv: "912831ccd513b66b387dc18885a83a12"
server_key_version: "encryptionkey@0"
I20260926 06:18:59.511497 19552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/ts-0/wal/instance:
uuid: "91cd62e43450456aadbf052b9e23c65c"
format_stamp: "Formatted at 2026-09-26 06:18:59 on dist-test-slave-hndd"
server_key: "539f9243aba64468be4925140c29f1fd"
server_key_iv: "912831ccd513b66b387dc18885a83a12"
server_key_version: "encryptionkey@0"
I20260926 06:18:59.514066 19552 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260926 06:18:59.515039 19567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:18:59.515317 19552 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:18:59.515466 19552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/ts-0/wal
uuid: "91cd62e43450456aadbf052b9e23c65c"
format_stamp: "Formatted at 2026-09-26 06:18:59 on dist-test-slave-hndd"
server_key: "539f9243aba64468be4925140c29f1fd"
server_key_iv: "912831ccd513b66b387dc18885a83a12"
server_key_version: "encryptionkey@0"
I20260926 06:18:59.515576 19552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:18:59.544725 19552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:18:59.546922 19552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:18:59.547086 19552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:18:59.547361 19552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:18:59.547717 19552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:18:59.547775 19552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:18:59.547812 19552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:18:59.547842 19552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:18:59.554171 19552 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.193:35639
I20260926 06:18:59.554227 19680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.193:35639 every 8 connection(s)
I20260926 06:18:59.554586 19552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
I20260926 06:18:59.554848 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 19552
I20260926 06:18:59.554955 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/ts-0/wal/instance
I20260926 06:18:59.555176 2491 external_mini_cluster.cc:1428] Setting key 79b5b869818c6e4294630f3e2603dbd7
Sep 26 06:18:59 dist-test-slave-hndd krb5kdc[19445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403539, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.193@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.110.194@KRBTEST.COM; defaulting to no policy
I20260926 06:18:59.566457 19544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:59.555565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:53047 (local address 127.2.110.254:42379)
0926 06:18:59.555638 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 06:18:59.555641 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:18:59.555814 (+ 173us) server_negotiation.cc:407] Connection header received
0926 06:18:59.556445 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:18:59.556446 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:18:59.556458 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:18:59.556500 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:18:59.557835 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:59.558549 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:59.559570 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:59.559782 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:59.559870 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:59.562638 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:18:59.562639 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:18:59.562640 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:18:59.562648 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:18:59.564174 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:18:59.564629 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:18:59.564630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:18:59.564631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:18:59.564692 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:18:59.564956 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:18:59.564957 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:18:59.564957 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:18:59.565109 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:18:59.565188 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 06:18:59.566029 (+ 841us) server_negotiation.cc:299] Negotiation successful
0926 06:18:59.566307 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 06:18:59.566572 19683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:59.555494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42379 (local address 127.2.110.193:53047)
0926 06:18:59.555682 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:59.555728 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 06:18:59.556335 (+ 607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:18:59.556580 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:18:59.556584 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:18:59.557397 (+ 813us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:18:59.557707 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:18:59.557714 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:18:59.558655 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:18:59.558656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:18:59.559466 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:18:59.559468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:18:59.559879 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:18:59.559880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:18:59.560022 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:59.560681 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:18:59.560691 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:18:59.562533 (+ 1842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:18:59.564284 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:18:59.564291 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:18:59.564293 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:18:59.564534 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:18:59.564759 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:18:59.564760 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:18:59.564761 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:18:59.564883 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:18:59.565178 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:18:59.565185 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:18:59.565672 (+ 487us) client_negotiation.cc:769] Sending connection context
0926 06:18:59.565894 (+ 222us) client_negotiation.cc:240] Negotiation successful
0926 06:18:59.566325 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":98,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.194@KRBTEST.COM" created.
I20260926 06:18:59.567142 19681 heartbeater.cc:347] Connected to a master server at 127.2.110.254:42379
I20260926 06:18:59.567238 19681 heartbeater.cc:464] Registering TS with master...
I20260926 06:18:59.567451 19681 heartbeater.cc:511] Master 127.2.110.254:42379 requested a full tablet report, sending...
I20260926 06:18:59.568017 19490 ts_manager.cc:194] Registered new tserver with Master: 91cd62e43450456aadbf052b9e23c65c (127.2.110.193:35639)
I20260926 06:18:59.568730 19490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.193@KRBTEST.COM'} at 127.2.110.193:53047
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab.
Entry for principal kudu/127.2.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab.
I20260926 06:18:59.585584 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.194:0
--local_ip_for_outbound_sockets=127.2.110.194
--webserver_interface=127.2.110.194
--webserver_port=0
--tserver_master_addrs=127.2.110.254:42379
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab
--principal=kudu/127.2.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:39825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:18:59.670538 19686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:18:59.670763 19686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:18:59.670789 19686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:18:59.671970 19686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:18:59.672008 19686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:18:59.672035 19686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.194
I20260926 06:18:59.673493 19686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:39825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab
--principal=kudu/127.2.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.110.194
--webserver_port=0
--tserver_master_addrs=127.2.110.254:42379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.194
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:18:59.673697 19686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:18:59.673889 19686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:18:59.678936 19691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:18:59.678961 19694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:18:59.678936 19692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:18:59.678963 19686 server_base.cc:1034] running on GCE node
I20260926 06:18:59.679320 19686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:18:59.679518 19686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:18:59.680668 19686 hybrid_clock.cc:648] HybridClock initialized: now 1790403539680657 us; error 28 us; skew 500 ppm
Sep 26 06:18:59 dist-test-slave-hndd krb5kdc[19445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403539, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:18:59.683171 19686 init.cc:377] Logged in from keytab as kudu/127.2.110.194@KRBTEST.COM (short username kudu)
I20260926 06:18:59.683653 19686 webserver.cc:460] Webserver started at http://127.2.110.194:42279/ using document root <none> and password file <none>
I20260926 06:18:59.683856 19686 fs_manager.cc:359] Metadata directory not provided
I20260926 06:18:59.683897 19686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:18:59.683988 19686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:18:59.684969 19686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/ts-1/data/instance:
uuid: "ddf888de57cb4a80945bc06388b8743a"
format_stamp: "Formatted at 2026-09-26 06:18:59 on dist-test-slave-hndd"
server_key: "7541db83451df1b17cc4061bddf6c443"
server_key_iv: "2d75ee1cffbf8f932dcf9691f3967791"
server_key_version: "encryptionkey@0"
I20260926 06:18:59.685494 19686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/ts-1/wal/instance:
uuid: "ddf888de57cb4a80945bc06388b8743a"
format_stamp: "Formatted at 2026-09-26 06:18:59 on dist-test-slave-hndd"
server_key: "7541db83451df1b17cc4061bddf6c443"
server_key_iv: "2d75ee1cffbf8f932dcf9691f3967791"
server_key_version: "encryptionkey@0"
I20260926 06:18:59.686872 19686 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:18:59.687637 19701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:18:59.687839 19686 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:18:59.687912 19686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/ts-1/wal
uuid: "ddf888de57cb4a80945bc06388b8743a"
format_stamp: "Formatted at 2026-09-26 06:18:59 on dist-test-slave-hndd"
server_key: "7541db83451df1b17cc4061bddf6c443"
server_key_iv: "2d75ee1cffbf8f932dcf9691f3967791"
server_key_version: "encryptionkey@0"
I20260926 06:18:59.687965 19686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:18:59.694958 19686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:18:59.697048 19686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:18:59.697166 19686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:18:59.697453 19686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:18:59.697732 19686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:18:59.697763 19686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:18:59.697788 19686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:18:59.697800 19686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:18:59.704324 19686 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.194:42337
I20260926 06:18:59.704365 19814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.194:42337 every 8 connection(s)
I20260926 06:18:59.704690 19686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
Sep 26 06:18:59 dist-test-slave-hndd krb5kdc[19445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403539, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.194@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:18:59.710883 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 19686
I20260926 06:18:59.711006 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/ts-1/wal/instance
I20260926 06:18:59.711264 2491 external_mini_cluster.cc:1428] Setting key 5f6bf1a96f37db9b56ee2c31f7dcee69
I20260926 06:18:59.714475 19544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:59.705477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:33555 (local address 127.2.110.254:42379)
0926 06:18:59.705543 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 06:18:59.705545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:18:59.705718 (+ 173us) server_negotiation.cc:407] Connection header received
0926 06:18:59.706203 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:18:59.706204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:18:59.706215 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:18:59.706282 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:18:59.707209 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:59.707716 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:59.708736 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:59.708962 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:59.709013 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:59.711156 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:18:59.711158 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:18:59.711159 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:18:59.711168 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:18:59.712757 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:18:59.713295 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:18:59.713296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:18:59.713296 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:18:59.713408 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:18:59.713707 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:18:59.713708 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:18:59.713708 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:18:59.713866 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:18:59.714143 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0926 06:18:59.714242 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 06:18:59.714346 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 06:18:59.714622 19817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:59.705455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42379 (local address 127.2.110.194:33555)
0926 06:18:59.705603 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:59.705644 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 06:18:59.706114 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:18:59.706333 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:18:59.706336 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:18:59.706907 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:18:59.707101 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:18:59.707106 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:18:59.707823 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:18:59.707824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:18:59.708650 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:18:59.708652 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:18:59.709030 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:18:59.709030 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:18:59.709125 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:59.709649 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:18:59.709657 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:18:59.711044 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:18:59.712850 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:18:59.712857 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:18:59.712859 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:18:59.713195 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:18:59.713486 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:18:59.713487 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:18:59.713487 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:18:59.713633 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:18:59.713965 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:18:59.713971 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:18:59.714113 (+ 142us) client_negotiation.cc:769] Sending connection context
0926 06:18:59.714213 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 06:18:59.714349 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":77,"threads_started":1}
I20260926 06:18:59.715217 19815 heartbeater.cc:347] Connected to a master server at 127.2.110.254:42379
I20260926 06:18:59.715332 19815 heartbeater.cc:464] Registering TS with master...
I20260926 06:18:59.715526 19815 heartbeater.cc:511] Master 127.2.110.254:42379 requested a full tablet report, sending...
I20260926 06:18:59.716003 19490 ts_manager.cc:194] Registered new tserver with Master: ddf888de57cb4a80945bc06388b8743a (127.2.110.194:42337)
I20260926 06:18:59.716559 19490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.194@KRBTEST.COM'} at 127.2.110.194:33555
WARNING: no policy specified for kudu/127.2.110.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab.
Entry for principal kudu/127.2.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab.
I20260926 06:18:59.742285 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.195:0
--local_ip_for_outbound_sockets=127.2.110.195
--webserver_interface=127.2.110.195
--webserver_port=0
--tserver_master_addrs=127.2.110.254:42379
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab
--principal=kudu/127.2.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:39825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:18:59.825309 19820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:18:59.825663 19820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:18:59.825703 19820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:18:59.827142 19820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:18:59.827199 19820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:18:59.827239 19820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.195
I20260926 06:18:59.828853 19820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:39825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab
--principal=kudu/127.2.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.110.195
--webserver_port=0
--tserver_master_addrs=127.2.110.254:42379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.195
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:18:59.829173 19820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:18:59.829489 19820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:18:59.834688 19825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:18:59.834697 19826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:18:59.834687 19828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:18:59.835009 19820 server_base.cc:1034] running on GCE node
I20260926 06:18:59.835260 19820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:18:59.835551 19820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:18:59.836733 19820 hybrid_clock.cc:648] HybridClock initialized: now 1790403539836702 us; error 45 us; skew 500 ppm
Sep 26 06:18:59 dist-test-slave-hndd krb5kdc[19445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403539, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:18:59.839494 19820 init.cc:377] Logged in from keytab as kudu/127.2.110.195@KRBTEST.COM (short username kudu)
I20260926 06:18:59.840054 19820 webserver.cc:460] Webserver started at http://127.2.110.195:36113/ using document root <none> and password file <none>
I20260926 06:18:59.840261 19820 fs_manager.cc:359] Metadata directory not provided
I20260926 06:18:59.840304 19820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:18:59.840394 19820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:18:59.841485 19820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/ts-2/data/instance:
uuid: "e2488b5fa0fa477eb5c8a6ddca77f7d2"
format_stamp: "Formatted at 2026-09-26 06:18:59 on dist-test-slave-hndd"
server_key: "b8444c8002d9f976cc3ef182f46122a8"
server_key_iv: "9568c5c4940342f1e102a0d9dd0ce0c6"
server_key_version: "encryptionkey@0"
I20260926 06:18:59.841804 19820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/ts-2/wal/instance:
uuid: "e2488b5fa0fa477eb5c8a6ddca77f7d2"
format_stamp: "Formatted at 2026-09-26 06:18:59 on dist-test-slave-hndd"
server_key: "b8444c8002d9f976cc3ef182f46122a8"
server_key_iv: "9568c5c4940342f1e102a0d9dd0ce0c6"
server_key_version: "encryptionkey@0"
I20260926 06:18:59.843176 19820 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:18:59.844007 19835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:18:59.844208 19820 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:18:59.844300 19820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/ts-2/wal
uuid: "e2488b5fa0fa477eb5c8a6ddca77f7d2"
format_stamp: "Formatted at 2026-09-26 06:18:59 on dist-test-slave-hndd"
server_key: "b8444c8002d9f976cc3ef182f46122a8"
server_key_iv: "9568c5c4940342f1e102a0d9dd0ce0c6"
server_key_version: "encryptionkey@0"
I20260926 06:18:59.844362 19820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:18:59.857250 19820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:18:59.859661 19820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:18:59.859833 19820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:18:59.860296 19820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:18:59.860694 19820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:18:59.860731 19820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:18:59.860755 19820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:18:59.860769 19820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:18:59.867390 19820 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.195:36233
I20260926 06:18:59.867472 19948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.195:36233 every 8 connection(s)
I20260926 06:18:59.867810 19820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
Sep 26 06:18:59 dist-test-slave-hndd krb5kdc[19445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403539, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.195@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:18:59.877868 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 19820
I20260926 06:18:59.877943 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/ts-2/wal/instance
I20260926 06:18:59.878110 2491 external_mini_cluster.cc:1428] Setting key 926e66aa28f3d35ce614dba8de4b0882
I20260926 06:18:59.880270 19951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:59.869162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:42379 (local address 127.2.110.195:51027)
0926 06:18:59.869439 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:59.869483 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 06:18:59.870156 (+ 673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:18:59.870458 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:18:59.870462 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:18:59.871172 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:18:59.871418 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:18:59.871422 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:18:59.872202 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:18:59.872203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:18:59.873080 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:18:59.873083 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:18:59.873577 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:18:59.873578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:18:59.873722 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:59.874473 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:18:59.874486 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:18:59.876284 (+ 1798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:18:59.878448 (+ 2164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:18:59.878458 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:18:59.878460 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:18:59.878886 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:18:59.879126 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:18:59.879127 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:18:59.879127 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:18:59.879281 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:18:59.879553 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:18:59.879559 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:18:59.879717 (+ 158us) client_negotiation.cc:769] Sending connection context
0926 06:18:59.879858 (+ 141us) client_negotiation.cc:240] Negotiation successful
0926 06:18:59.880081 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":130,"threads_started":1}
I20260926 06:18:59.880302 19544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:59.869179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:51027 (local address 127.2.110.254:42379)
0926 06:18:59.869278 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0926 06:18:59.869282 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:18:59.869602 (+ 320us) server_negotiation.cc:407] Connection header received
0926 06:18:59.870281 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:18:59.870283 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:18:59.870298 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:18:59.870368 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:18:59.871526 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:59.872100 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:59.873192 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:59.873476 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:59.873571 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:59.876495 (+ 2924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:18:59.876498 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:18:59.876498 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:18:59.876507 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:18:59.878340 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:18:59.878966 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:18:59.878967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:18:59.878968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:18:59.879044 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:18:59.879343 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:18:59.879344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:18:59.879344 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:18:59.879481 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:18:59.879561 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 06:18:59.880006 (+ 445us) server_negotiation.cc:299] Negotiation successful
0926 06:18:59.880211 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 06:18:59.880817 19949 heartbeater.cc:347] Connected to a master server at 127.2.110.254:42379
I20260926 06:18:59.880908 19949 heartbeater.cc:464] Registering TS with master...
I20260926 06:18:59.881062 19949 heartbeater.cc:511] Master 127.2.110.254:42379 requested a full tablet report, sending...
I20260926 06:18:59.881435 19490 ts_manager.cc:194] Registered new tserver with Master: e2488b5fa0fa477eb5c8a6ddca77f7d2 (127.2.110.195:36233)
I20260926 06:18:59.881843 19490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.195@KRBTEST.COM'} at 127.2.110.195:51027
I20260926 06:18:59.890295 2491 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 06:18:59.898581 19544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:59.892052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50688 (local address 127.2.110.254:42379)
0926 06:18:59.892146 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0926 06:18:59.892149 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:18:59.892281 (+ 132us) server_negotiation.cc:407] Connection header received
0926 06:18:59.892311 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:18:59.892311 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:18:59.892324 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:18:59.892384 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:18:59.892953 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:59.893558 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:59.894294 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:59.894512 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:59.894585 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:59.895878 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:18:59.895879 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:18:59.895879 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:18:59.895886 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:18:59.897318 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:18:59.897711 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:18:59.897712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:18:59.897712 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:18:59.897764 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:18:59.897972 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:18:59.897973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:18:59.897980 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:18:59.898151 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:18:59.898241 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 06:18:59.898431 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 06:18:59.898512 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 06:18:59.901117 19490 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50688:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:18:59.903077 19490 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:18:59.911206 19960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:59.906771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.194:42337 (local address 127.0.0.1:54126)
0926 06:18:59.907098 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:59.907112 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 06:18:59.907170 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:18:59.907675 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:18:59.907675 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:18:59.907678 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:18:59.907730 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:18:59.907731 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:18:59.908650 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:18:59.908651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:18:59.909964 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:18:59.909966 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:18:59.910851 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:18:59.910852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:18:59.910945 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:59.910959 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 06:18:59.911062 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 06:18:59.911108 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"spinlock_wait_cycles":896,"thread_start_us":126,"threads_started":1}
I20260926 06:18:59.911217 19963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:59.906964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54126 (local address 127.2.110.194:42337)
0926 06:18:59.907499 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0926 06:18:59.907504 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:18:59.907515 (+ 11us) server_negotiation.cc:407] Connection header received
0926 06:18:59.907555 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:18:59.907558 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:18:59.907573 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:18:59.907804 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:18:59.907858 (+ 54us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:59.908542 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:59.910060 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:59.910758 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:59.910850 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:59.910914 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 06:18:59.911082 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 06:18:59.911131 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":96,"threads_started":1}
I20260926 06:18:59.911425 19964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:59.907055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53886 (local address 127.2.110.193:35639)
0926 06:18:59.907377 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0926 06:18:59.907381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:18:59.907390 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:18:59.907425 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:18:59.907428 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:18:59.907441 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:18:59.907728 (+ 287us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:18:59.907792 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:59.908859 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:59.910176 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:59.910795 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:59.911161 (+ 366us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:59.911219 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 06:18:59.911248 (+ 29us) server_negotiation.cc:299] Negotiation successful
0926 06:18:59.911299 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":94,"threads_started":1}
I20260926 06:18:59.911660 19962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:59.906767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.193:35639 (local address 127.0.0.1:53886)
0926 06:18:59.906975 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:59.907019 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 06:18:59.907089 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:18:59.907552 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:18:59.907555 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:18:59.907557 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:18:59.907642 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:18:59.907644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:18:59.908994 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:18:59.908995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:18:59.910079 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:18:59.910083 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:18:59.910896 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:18:59.910897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:18:59.910971 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:59.910976 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 06:18:59.911011 (+ 35us) client_negotiation.cc:240] Negotiation successful
0926 06:18:59.911072 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":3,"thread_start_us":69,"threads_started":1}
I20260926 06:18:59.912425 19749 tablet_service.cc:1467] Processing CreateTablet for tablet 42ff2fd67a994588aa79f564f36d5a04 (DEFAULT_TABLE table=test-table [id=b95f76ed09a348c3b1146916512c58aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:18:59.912559 19615 tablet_service.cc:1467] Processing CreateTablet for tablet 42ff2fd67a994588aa79f564f36d5a04 (DEFAULT_TABLE table=test-table [id=b95f76ed09a348c3b1146916512c58aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:18:59.912948 19615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42ff2fd67a994588aa79f564f36d5a04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:18:59.912964 19749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42ff2fd67a994588aa79f564f36d5a04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:18:59.913883 19961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:59.906767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.195:36233 (local address 127.0.0.1:56870)
0926 06:18:59.907264 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:59.907271 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 06:18:59.907299 (+ 28us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:18:59.907584 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:18:59.907585 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:18:59.907586 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:18:59.907642 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:18:59.907644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:18:59.909431 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:18:59.909433 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:18:59.910438 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:18:59.910441 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:18:59.913609 (+ 3168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:18:59.913610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:18:59.913680 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:59.913685 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 06:18:59.913763 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 06:18:59.913809 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"mutex_wait_us":9,"thread_start_us":84,"threads_started":1}
I20260926 06:18:59.914162 19965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:59.907148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56870 (local address 127.2.110.195:36233)
0926 06:18:59.907329 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 06:18:59.907335 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 06:18:59.907347 (+ 12us) server_negotiation.cc:407] Connection header received
0926 06:18:59.907398 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:18:59.907409 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:18:59.907426 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:18:59.907510 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:18:59.907723 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:59.908877 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:59.910540 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:59.911248 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:59.911999 (+ 751us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:59.912113 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 06:18:59.913860 (+ 1747us) server_negotiation.cc:299] Negotiation successful
0926 06:18:59.913935 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":88,"threads_started":1}
I20260926 06:18:59.916333 19966 tablet_bootstrap.cc:492] T 42ff2fd67a994588aa79f564f36d5a04 P ddf888de57cb4a80945bc06388b8743a: Bootstrap starting.
I20260926 06:18:59.916917 19966 tablet_bootstrap.cc:654] T 42ff2fd67a994588aa79f564f36d5a04 P ddf888de57cb4a80945bc06388b8743a: Neither blocks nor log segments found. Creating new log.
I20260926 06:18:59.917198 19967 tablet_bootstrap.cc:492] T 42ff2fd67a994588aa79f564f36d5a04 P 91cd62e43450456aadbf052b9e23c65c: Bootstrap starting.
I20260926 06:18:59.917573 19966 log.cc:824] T 42ff2fd67a994588aa79f564f36d5a04 P ddf888de57cb4a80945bc06388b8743a: Log is configured to *not* fsync() on all Append() calls
I20260926 06:18:59.917821 19883 tablet_service.cc:1467] Processing CreateTablet for tablet 42ff2fd67a994588aa79f564f36d5a04 (DEFAULT_TABLE table=test-table [id=b95f76ed09a348c3b1146916512c58aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:18:59.917958 19967 tablet_bootstrap.cc:654] T 42ff2fd67a994588aa79f564f36d5a04 P 91cd62e43450456aadbf052b9e23c65c: Neither blocks nor log segments found. Creating new log.
I20260926 06:18:59.918179 19883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42ff2fd67a994588aa79f564f36d5a04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:18:59.918570 19966 tablet_bootstrap.cc:492] T 42ff2fd67a994588aa79f564f36d5a04 P ddf888de57cb4a80945bc06388b8743a: No bootstrap required, opened a new log
I20260926 06:18:59.918536 19967 log.cc:824] T 42ff2fd67a994588aa79f564f36d5a04 P 91cd62e43450456aadbf052b9e23c65c: Log is configured to *not* fsync() on all Append() calls
I20260926 06:18:59.918640 19966 ts_tablet_manager.cc:1397] T 42ff2fd67a994588aa79f564f36d5a04 P ddf888de57cb4a80945bc06388b8743a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 06:18:59.920462 19966 raft_consensus.cc:348] T 42ff2fd67a994588aa79f564f36d5a04 P ddf888de57cb4a80945bc06388b8743a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2488b5fa0fa477eb5c8a6ddca77f7d2" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36233 } } peers { permanent_uuid: "ddf888de57cb4a80945bc06388b8743a" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 42337 } } peers { permanent_uuid: "91cd62e43450456aadbf052b9e23c65c" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 35639 } }
I20260926 06:18:59.920617 19966 raft_consensus.cc:374] T 42ff2fd67a994588aa79f564f36d5a04 P ddf888de57cb4a80945bc06388b8743a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:18:59.920640 19966 raft_consensus.cc:729] T 42ff2fd67a994588aa79f564f36d5a04 P ddf888de57cb4a80945bc06388b8743a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddf888de57cb4a80945bc06388b8743a, State: Initialized, Role: FOLLOWER
I20260926 06:18:59.920771 19966 consensus_queue.cc:260] T 42ff2fd67a994588aa79f564f36d5a04 P ddf888de57cb4a80945bc06388b8743a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2488b5fa0fa477eb5c8a6ddca77f7d2" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36233 } } peers { permanent_uuid: "ddf888de57cb4a80945bc06388b8743a" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 42337 } } peers { permanent_uuid: "91cd62e43450456aadbf052b9e23c65c" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 35639 } }
I20260926 06:18:59.921100 19966 ts_tablet_manager.cc:1428] T 42ff2fd67a994588aa79f564f36d5a04 P ddf888de57cb4a80945bc06388b8743a: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260926 06:18:59.921124 19815 heartbeater.cc:503] Master 127.2.110.254:42379 was elected leader, sending a full tablet report...
I20260926 06:18:59.921197 19971 tablet_bootstrap.cc:492] T 42ff2fd67a994588aa79f564f36d5a04 P e2488b5fa0fa477eb5c8a6ddca77f7d2: Bootstrap starting.
I20260926 06:18:59.921810 19967 tablet_bootstrap.cc:492] T 42ff2fd67a994588aa79f564f36d5a04 P 91cd62e43450456aadbf052b9e23c65c: No bootstrap required, opened a new log
I20260926 06:18:59.921892 19967 ts_tablet_manager.cc:1397] T 42ff2fd67a994588aa79f564f36d5a04 P 91cd62e43450456aadbf052b9e23c65c: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260926 06:18:59.921940 19971 tablet_bootstrap.cc:654] T 42ff2fd67a994588aa79f564f36d5a04 P e2488b5fa0fa477eb5c8a6ddca77f7d2: Neither blocks nor log segments found. Creating new log.
I20260926 06:18:59.922516 19971 log.cc:824] T 42ff2fd67a994588aa79f564f36d5a04 P e2488b5fa0fa477eb5c8a6ddca77f7d2: Log is configured to *not* fsync() on all Append() calls
I20260926 06:18:59.923280 19971 tablet_bootstrap.cc:492] T 42ff2fd67a994588aa79f564f36d5a04 P e2488b5fa0fa477eb5c8a6ddca77f7d2: No bootstrap required, opened a new log
I20260926 06:18:59.923354 19971 ts_tablet_manager.cc:1397] T 42ff2fd67a994588aa79f564f36d5a04 P e2488b5fa0fa477eb5c8a6ddca77f7d2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:18:59.924163 19967 raft_consensus.cc:348] T 42ff2fd67a994588aa79f564f36d5a04 P 91cd62e43450456aadbf052b9e23c65c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2488b5fa0fa477eb5c8a6ddca77f7d2" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36233 } } peers { permanent_uuid: "ddf888de57cb4a80945bc06388b8743a" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 42337 } } peers { permanent_uuid: "91cd62e43450456aadbf052b9e23c65c" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 35639 } }
I20260926 06:18:59.924371 19967 raft_consensus.cc:374] T 42ff2fd67a994588aa79f564f36d5a04 P 91cd62e43450456aadbf052b9e23c65c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:18:59.924408 19967 raft_consensus.cc:729] T 42ff2fd67a994588aa79f564f36d5a04 P 91cd62e43450456aadbf052b9e23c65c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91cd62e43450456aadbf052b9e23c65c, State: Initialized, Role: FOLLOWER
I20260926 06:18:59.924533 19967 consensus_queue.cc:260] T 42ff2fd67a994588aa79f564f36d5a04 P 91cd62e43450456aadbf052b9e23c65c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2488b5fa0fa477eb5c8a6ddca77f7d2" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36233 } } peers { permanent_uuid: "ddf888de57cb4a80945bc06388b8743a" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 42337 } } peers { permanent_uuid: "91cd62e43450456aadbf052b9e23c65c" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 35639 } }
I20260926 06:18:59.924825 19967 ts_tablet_manager.cc:1428] T 42ff2fd67a994588aa79f564f36d5a04 P 91cd62e43450456aadbf052b9e23c65c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 06:18:59.924868 19681 heartbeater.cc:503] Master 127.2.110.254:42379 was elected leader, sending a full tablet report...
I20260926 06:18:59.925283 19971 raft_consensus.cc:348] T 42ff2fd67a994588aa79f564f36d5a04 P e2488b5fa0fa477eb5c8a6ddca77f7d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2488b5fa0fa477eb5c8a6ddca77f7d2" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36233 } } peers { permanent_uuid: "ddf888de57cb4a80945bc06388b8743a" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 42337 } } peers { permanent_uuid: "91cd62e43450456aadbf052b9e23c65c" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 35639 } }
I20260926 06:18:59.925519 19971 raft_consensus.cc:374] T 42ff2fd67a994588aa79f564f36d5a04 P e2488b5fa0fa477eb5c8a6ddca77f7d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:18:59.925549 19971 raft_consensus.cc:729] T 42ff2fd67a994588aa79f564f36d5a04 P e2488b5fa0fa477eb5c8a6ddca77f7d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2488b5fa0fa477eb5c8a6ddca77f7d2, State: Initialized, Role: FOLLOWER
I20260926 06:18:59.925664 19971 consensus_queue.cc:260] T 42ff2fd67a994588aa79f564f36d5a04 P e2488b5fa0fa477eb5c8a6ddca77f7d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2488b5fa0fa477eb5c8a6ddca77f7d2" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36233 } } peers { permanent_uuid: "ddf888de57cb4a80945bc06388b8743a" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 42337 } } peers { permanent_uuid: "91cd62e43450456aadbf052b9e23c65c" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 35639 } }
I20260926 06:18:59.925976 19971 ts_tablet_manager.cc:1428] T 42ff2fd67a994588aa79f564f36d5a04 P e2488b5fa0fa477eb5c8a6ddca77f7d2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 06:18:59.925983 19949 heartbeater.cc:503] Master 127.2.110.254:42379 was elected leader, sending a full tablet report...
W20260926 06:18:59.955272 19816 tablet.cc:2369] T 42ff2fd67a994588aa79f564f36d5a04 P ddf888de57cb4a80945bc06388b8743a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:18:59.994484 19974 raft_consensus.cc:482] T 42ff2fd67a994588aa79f564f36d5a04 P e2488b5fa0fa477eb5c8a6ddca77f7d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:18:59.994773 19974 raft_consensus.cc:504] T 42ff2fd67a994588aa79f564f36d5a04 P e2488b5fa0fa477eb5c8a6ddca77f7d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2488b5fa0fa477eb5c8a6ddca77f7d2" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36233 } } peers { permanent_uuid: "ddf888de57cb4a80945bc06388b8743a" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 42337 } } peers { permanent_uuid: "91cd62e43450456aadbf052b9e23c65c" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 35639 } }
I20260926 06:18:59.995252 19974 leader_election.cc:288] T 42ff2fd67a994588aa79f564f36d5a04 P e2488b5fa0fa477eb5c8a6ddca77f7d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddf888de57cb4a80945bc06388b8743a (127.2.110.194:42337), 91cd62e43450456aadbf052b9e23c65c (127.2.110.193:35639)
I20260926 06:18:59.998266 19963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:59.995604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:53549 (local address 127.2.110.194:42337)
0926 06:18:59.995682 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 06:18:59.995685 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:18:59.995696 (+ 11us) server_negotiation.cc:407] Connection header received
0926 06:18:59.995734 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:18:59.995736 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:18:59.995744 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:18:59.995839 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:18:59.996014 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:59.996504 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:59.997495 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:59.997890 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:59.997976 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:59.997987 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 06:18:59.998154 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 06:18:59.998196 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 06:18:59.998266 19951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:59.995488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.194:42337 (local address 127.2.110.195:53549)
0926 06:18:59.995559 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:59.995575 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 06:18:59.995629 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:18:59.995853 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:18:59.995854 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:18:59.995857 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:18:59.995936 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:18:59.995938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:18:59.996594 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:18:59.996595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:18:59.997396 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:18:59.997398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:18:59.997983 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:18:59.997984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:18:59.998066 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:59.998069 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:18:59.998146 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 06:18:59.998190 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 06:18:59.998457 19975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:59.995491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.193:35639 (local address 127.2.110.195:36387)
0926 06:18:59.995812 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0926 06:18:59.995822 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:18:59.995905 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:18:59.996061 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:18:59.996062 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:18:59.996063 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:18:59.996136 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:18:59.996138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:18:59.996771 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:18:59.996772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:18:59.997625 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:18:59.997627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:18:59.998188 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:18:59.998188 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:18:59.998265 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:59.998269 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:18:59.998332 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 06:18:59.998375 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":148,"thread_start_us":121,"threads_started":1}
I20260926 06:18:59.998469 19964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:18:59.995669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:36387 (local address 127.2.110.193:35639)
0926 06:18:59.995729 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 06:18:59.995731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:18:59.995899 (+ 168us) server_negotiation.cc:407] Connection header received
0926 06:18:59.995964 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:18:59.995965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:18:59.995982 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:18:59.996059 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:18:59.996217 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:59.996684 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:59.997716 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:18:59.998097 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:18:59.998188 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:18:59.998196 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 06:18:59.998350 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 06:18:59.998391 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 06:18:59.998823 19769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ff2fd67a994588aa79f564f36d5a04" candidate_uuid: "e2488b5fa0fa477eb5c8a6ddca77f7d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddf888de57cb4a80945bc06388b8743a" is_pre_election: true
I20260926 06:18:59.999012 19769 raft_consensus.cc:2454] T 42ff2fd67a994588aa79f564f36d5a04 P ddf888de57cb4a80945bc06388b8743a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2488b5fa0fa477eb5c8a6ddca77f7d2 in term 0.
I20260926 06:18:59.998986 19635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ff2fd67a994588aa79f564f36d5a04" candidate_uuid: "e2488b5fa0fa477eb5c8a6ddca77f7d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91cd62e43450456aadbf052b9e23c65c" is_pre_election: true
I20260926 06:18:59.999164 19635 raft_consensus.cc:2454] T 42ff2fd67a994588aa79f564f36d5a04 P 91cd62e43450456aadbf052b9e23c65c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2488b5fa0fa477eb5c8a6ddca77f7d2 in term 0.
I20260926 06:18:59.999259 19839 leader_election.cc:302] T 42ff2fd67a994588aa79f564f36d5a04 P e2488b5fa0fa477eb5c8a6ddca77f7d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ddf888de57cb4a80945bc06388b8743a, e2488b5fa0fa477eb5c8a6ddca77f7d2; no voters:
I20260926 06:18:59.999408 19974 raft_consensus.cc:2781] T 42ff2fd67a994588aa79f564f36d5a04 P e2488b5fa0fa477eb5c8a6ddca77f7d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:18:59.999460 19974 raft_consensus.cc:482] T 42ff2fd67a994588aa79f564f36d5a04 P e2488b5fa0fa477eb5c8a6ddca77f7d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:18:59.999477 19974 raft_consensus.cc:3037] T 42ff2fd67a994588aa79f564f36d5a04 P e2488b5fa0fa477eb5c8a6ddca77f7d2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:19:00.000172 19974 raft_consensus.cc:504] T 42ff2fd67a994588aa79f564f36d5a04 P e2488b5fa0fa477eb5c8a6ddca77f7d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2488b5fa0fa477eb5c8a6ddca77f7d2" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36233 } } peers { permanent_uuid: "ddf888de57cb4a80945bc06388b8743a" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 42337 } } peers { permanent_uuid: "91cd62e43450456aadbf052b9e23c65c" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 35639 } }
I20260926 06:19:00.000357 19974 leader_election.cc:288] T 42ff2fd67a994588aa79f564f36d5a04 P e2488b5fa0fa477eb5c8a6ddca77f7d2 [CANDIDATE]: Term 1 election: Requested vote from peers ddf888de57cb4a80945bc06388b8743a (127.2.110.194:42337), 91cd62e43450456aadbf052b9e23c65c (127.2.110.193:35639)
I20260926 06:19:00.000523 19769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ff2fd67a994588aa79f564f36d5a04" candidate_uuid: "e2488b5fa0fa477eb5c8a6ddca77f7d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddf888de57cb4a80945bc06388b8743a"
I20260926 06:19:00.000530 19635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ff2fd67a994588aa79f564f36d5a04" candidate_uuid: "e2488b5fa0fa477eb5c8a6ddca77f7d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91cd62e43450456aadbf052b9e23c65c"
I20260926 06:19:00.000614 19769 raft_consensus.cc:3037] T 42ff2fd67a994588aa79f564f36d5a04 P ddf888de57cb4a80945bc06388b8743a [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:19:00.000634 19635 raft_consensus.cc:3037] T 42ff2fd67a994588aa79f564f36d5a04 P 91cd62e43450456aadbf052b9e23c65c [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:19:00.001426 19769 raft_consensus.cc:2454] T 42ff2fd67a994588aa79f564f36d5a04 P ddf888de57cb4a80945bc06388b8743a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2488b5fa0fa477eb5c8a6ddca77f7d2 in term 1.
I20260926 06:19:00.001436 19635 raft_consensus.cc:2454] T 42ff2fd67a994588aa79f564f36d5a04 P 91cd62e43450456aadbf052b9e23c65c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2488b5fa0fa477eb5c8a6ddca77f7d2 in term 1.
I20260926 06:19:00.001605 19837 leader_election.cc:302] T 42ff2fd67a994588aa79f564f36d5a04 P e2488b5fa0fa477eb5c8a6ddca77f7d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91cd62e43450456aadbf052b9e23c65c, e2488b5fa0fa477eb5c8a6ddca77f7d2; no voters:
I20260926 06:19:00.001698 19974 raft_consensus.cc:2781] T 42ff2fd67a994588aa79f564f36d5a04 P e2488b5fa0fa477eb5c8a6ddca77f7d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:19:00.001855 19974 raft_consensus.cc:686] T 42ff2fd67a994588aa79f564f36d5a04 P e2488b5fa0fa477eb5c8a6ddca77f7d2 [term 1 LEADER]: Becoming Leader. State: Replica: e2488b5fa0fa477eb5c8a6ddca77f7d2, State: Running, Role: LEADER
I20260926 06:19:00.001933 19974 consensus_queue.cc:237] T 42ff2fd67a994588aa79f564f36d5a04 P e2488b5fa0fa477eb5c8a6ddca77f7d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2488b5fa0fa477eb5c8a6ddca77f7d2" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36233 } } peers { permanent_uuid: "ddf888de57cb4a80945bc06388b8743a" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 42337 } } peers { permanent_uuid: "91cd62e43450456aadbf052b9e23c65c" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 35639 } }
I20260926 06:19:00.003206 19490 catalog_manager.cc:5495] T 42ff2fd67a994588aa79f564f36d5a04 P e2488b5fa0fa477eb5c8a6ddca77f7d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to e2488b5fa0fa477eb5c8a6ddca77f7d2 (127.2.110.195). New cstate: current_term: 1 leader_uuid: "e2488b5fa0fa477eb5c8a6ddca77f7d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2488b5fa0fa477eb5c8a6ddca77f7d2" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 36233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddf888de57cb4a80945bc06388b8743a" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 42337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91cd62e43450456aadbf052b9e23c65c" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 35639 } health_report { overall_health: UNKNOWN } } }
I20260926 06:19:00.029587 19965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:00.026902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56874 (local address 127.2.110.195:36233)
0926 06:19:00.026992 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:00.026994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:00.027004 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:19:00.027043 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:00.027044 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:00.027063 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:00.027139 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:19:00.027266 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:00.027773 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:00.028743 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:00.028952 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:00.029054 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:00.029195 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:19:00.029304 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:19:00.029393 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:00.029446 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 06:19:00.029466 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 06:19:00.031706 19769 raft_consensus.cc:1261] T 42ff2fd67a994588aa79f564f36d5a04 P ddf888de57cb4a80945bc06388b8743a [term 1 FOLLOWER]: Refusing update from remote peer e2488b5fa0fa477eb5c8a6ddca77f7d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:19:00.031711 19635 raft_consensus.cc:1261] T 42ff2fd67a994588aa79f564f36d5a04 P 91cd62e43450456aadbf052b9e23c65c [term 1 FOLLOWER]: Refusing update from remote peer e2488b5fa0fa477eb5c8a6ddca77f7d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:19:00.032074 19974 consensus_queue.cc:1035] T 42ff2fd67a994588aa79f564f36d5a04 P e2488b5fa0fa477eb5c8a6ddca77f7d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91cd62e43450456aadbf052b9e23c65c" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 35639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:19:00.032228 19976 consensus_queue.cc:1035] T 42ff2fd67a994588aa79f564f36d5a04 P e2488b5fa0fa477eb5c8a6ddca77f7d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddf888de57cb4a80945bc06388b8743a" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 42337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:19:00.034924 19980 mvcc.cc:205] Tried to move back new op lower bound from 7333492899966935040 to 7333492899848941568. Current Snapshot: MvccSnapshot[applied={T|T < 7333492899966935040}]
I20260926 06:19:00.036513 19490 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:19:00.036630 19490 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:19:00.037540 19490 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fa5cb55d4a8b4139a7fdc9775dd6916f: Sending DeleteTablet for 3 replicas of tablet 42ff2fd67a994588aa79f564f36d5a04
I20260926 06:19:00.037879 19883 tablet_service.cc:1514] Processing DeleteTablet for tablet 42ff2fd67a994588aa79f564f36d5a04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:19:00 UTC) from {username='kudu', principal='kudu/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:56870
I20260926 06:19:00.037904 19749 tablet_service.cc:1514] Processing DeleteTablet for tablet 42ff2fd67a994588aa79f564f36d5a04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:19:00 UTC) from {username='kudu', principal='kudu/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:54126
I20260926 06:19:00.037987 19615 tablet_service.cc:1514] Processing DeleteTablet for tablet 42ff2fd67a994588aa79f564f36d5a04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:19:00 UTC) from {username='kudu', principal='kudu/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:53886
I20260926 06:19:00.038252 19991 tablet_replica.cc:330] T 42ff2fd67a994588aa79f564f36d5a04 P 91cd62e43450456aadbf052b9e23c65c: stopping tablet replica
I20260926 06:19:00.038336 19989 tablet_replica.cc:330] T 42ff2fd67a994588aa79f564f36d5a04 P e2488b5fa0fa477eb5c8a6ddca77f7d2: stopping tablet replica
I20260926 06:19:00.038425 19989 raft_consensus.cc:2229] T 42ff2fd67a994588aa79f564f36d5a04 P e2488b5fa0fa477eb5c8a6ddca77f7d2 [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:19:00.038483 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 19552
I20260926 06:19:00.039129 19989 raft_consensus.cc:2258] T 42ff2fd67a994588aa79f564f36d5a04 P e2488b5fa0fa477eb5c8a6ddca77f7d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:19:00.039594 19989 ts_tablet_manager.cc:1905] T 42ff2fd67a994588aa79f564f36d5a04 P e2488b5fa0fa477eb5c8a6ddca77f7d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:19:00.040872 19990 tablet_replica.cc:330] T 42ff2fd67a994588aa79f564f36d5a04 P ddf888de57cb4a80945bc06388b8743a: stopping tablet replica
I20260926 06:19:00.040992 19990 raft_consensus.cc:2229] T 42ff2fd67a994588aa79f564f36d5a04 P ddf888de57cb4a80945bc06388b8743a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:19:00.041121 19990 raft_consensus.cc:2258] T 42ff2fd67a994588aa79f564f36d5a04 P ddf888de57cb4a80945bc06388b8743a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:19:00.041615 19989 ts_tablet_manager.cc:1918] T 42ff2fd67a994588aa79f564f36d5a04 P e2488b5fa0fa477eb5c8a6ddca77f7d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:19:00.041738 19989 log.cc:1196] T 42ff2fd67a994588aa79f564f36d5a04 P e2488b5fa0fa477eb5c8a6ddca77f7d2: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/ts-2/wal/wals/42ff2fd67a994588aa79f564f36d5a04
I20260926 06:19:00.042078 19989 ts_tablet_manager.cc:1939] T 42ff2fd67a994588aa79f564f36d5a04 P e2488b5fa0fa477eb5c8a6ddca77f7d2: Deleting consensus metadata
I20260926 06:19:00.042552 19476 catalog_manager.cc:4841] TS e2488b5fa0fa477eb5c8a6ddca77f7d2 (127.2.110.195:36233): tablet 42ff2fd67a994588aa79f564f36d5a04 (table test-table [id=b95f76ed09a348c3b1146916512c58aa]) successfully deleted
W20260926 06:19:00.044557 19475 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.110.193:35639 (error 108)
I20260926 06:19:00.045251 19990 ts_tablet_manager.cc:1905] T 42ff2fd67a994588aa79f564f36d5a04 P ddf888de57cb4a80945bc06388b8743a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:19:00.047050 19990 ts_tablet_manager.cc:1918] T 42ff2fd67a994588aa79f564f36d5a04 P ddf888de57cb4a80945bc06388b8743a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:19:00.047201 19990 log.cc:1196] T 42ff2fd67a994588aa79f564f36d5a04 P ddf888de57cb4a80945bc06388b8743a: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403292369990-2491-0/minicluster-data/ts-1/wal/wals/42ff2fd67a994588aa79f564f36d5a04
I20260926 06:19:00.047565 19990 ts_tablet_manager.cc:1939] T 42ff2fd67a994588aa79f564f36d5a04 P ddf888de57cb4a80945bc06388b8743a: Deleting consensus metadata
W20260926 06:19:00.048264 19961 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:19:00.044987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.193:35639 (local address 127.0.0.1:53902)
0926 06:19:00.045027 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:00.045040 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:00.045081 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:00.048205 (+ 3124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.193:35639: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":20}
I20260926 06:19:00.048292 19477 catalog_manager.cc:4841] TS ddf888de57cb4a80945bc06388b8743a (127.2.110.194:42337): tablet 42ff2fd67a994588aa79f564f36d5a04 (table test-table [id=b95f76ed09a348c3b1146916512c58aa]) successfully deleted
W20260926 06:19:00.048395 19475 catalog_manager.cc:4584] TS 91cd62e43450456aadbf052b9e23c65c (127.2.110.193:35639): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 42ff2fd67a994588aa79f564f36d5a04: Network error: Client connection negotiation failed: client connection to 127.2.110.193:35639: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 06:19:00.048497 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 19686
I20260926 06:19:00.055925 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 19820
I20260926 06:19:00.060909 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 19458
2026-09-26T06:19:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2047 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T06:19:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:19:00Z Disabled control of system clock
I20260926 06:19:00.090431 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:39677
--webserver_interface=127.2.110.254
--webserver_port=0
--builtin_ntp_servers=127.2.110.212:37611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.110.254:39677
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 06:19:00.174283 19996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:19:00.174533 19996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:19:00.174556 19996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:19:00.175900 19996 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:19:00.175959 19996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:19:00.175974 19996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:19:00.175989 19996 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:19:00.176002 19996 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:19:00.177465 19996 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:37611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.110.254:39677
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:39677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:19:00.177695 19996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:19:00.177958 19996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:19:00.182966 20004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:00.183130 20001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:00.182966 20002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:00.183372 19996 server_base.cc:1034] running on GCE node
I20260926 06:19:00.183646 19996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:19:00.183960 19996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:19:00.185145 19996 hybrid_clock.cc:648] HybridClock initialized: now 1790403540185096 us; error 66 us; skew 500 ppm
I20260926 06:19:00.186650 19996 webserver.cc:460] Webserver started at http://127.2.110.254:43907/ using document root <none> and password file <none>
I20260926 06:19:00.186966 19996 fs_manager.cc:359] Metadata directory not provided
I20260926 06:19:00.187260 19996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:19:00.187453 19996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:19:00.188616 19996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/master-0/data/instance:
uuid: "08189ed1a4f44f898d9c1c6f33bbdc3d"
format_stamp: "Formatted at 2026-09-26 06:19:00 on dist-test-slave-hndd"
server_key: "85825cbfd1a34b6a31b33fe1b01170e0"
server_key_iv: "6bd9f7c7d956ab5e7861cbe36fd2734b"
server_key_version: "encryptionkey@0"
I20260926 06:19:00.188997 19996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/master-0/wal/instance:
uuid: "08189ed1a4f44f898d9c1c6f33bbdc3d"
format_stamp: "Formatted at 2026-09-26 06:19:00 on dist-test-slave-hndd"
server_key: "85825cbfd1a34b6a31b33fe1b01170e0"
server_key_iv: "6bd9f7c7d956ab5e7861cbe36fd2734b"
server_key_version: "encryptionkey@0"
I20260926 06:19:00.190560 19996 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260926 06:19:00.191522 20010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:00.191762 19996 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:19:00.191898 19996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "08189ed1a4f44f898d9c1c6f33bbdc3d"
format_stamp: "Formatted at 2026-09-26 06:19:00 on dist-test-slave-hndd"
server_key: "85825cbfd1a34b6a31b33fe1b01170e0"
server_key_iv: "6bd9f7c7d956ab5e7861cbe36fd2734b"
server_key_version: "encryptionkey@0"
I20260926 06:19:00.191996 19996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:19:00.212251 19996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:19:00.212654 19996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:19:00.212827 19996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:19:00.217269 19996 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.254:39677
I20260926 06:19:00.217314 20062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.254:39677 every 8 connection(s)
I20260926 06:19:00.217723 19996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:19:00.218713 20063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:19:00.220986 20063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08189ed1a4f44f898d9c1c6f33bbdc3d: Bootstrap starting.
I20260926 06:19:00.221640 20063 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08189ed1a4f44f898d9c1c6f33bbdc3d: Neither blocks nor log segments found. Creating new log.
I20260926 06:19:00.221889 20063 log.cc:824] T 00000000000000000000000000000000 P 08189ed1a4f44f898d9c1c6f33bbdc3d: Log is configured to *not* fsync() on all Append() calls
I20260926 06:19:00.222819 20063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08189ed1a4f44f898d9c1c6f33bbdc3d: No bootstrap required, opened a new log
I20260926 06:19:00.224643 20063 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08189ed1a4f44f898d9c1c6f33bbdc3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08189ed1a4f44f898d9c1c6f33bbdc3d" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 39677 } }
I20260926 06:19:00.224815 20063 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08189ed1a4f44f898d9c1c6f33bbdc3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:19:00.224843 20063 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08189ed1a4f44f898d9c1c6f33bbdc3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08189ed1a4f44f898d9c1c6f33bbdc3d, State: Initialized, Role: FOLLOWER
I20260926 06:19:00.225013 20063 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08189ed1a4f44f898d9c1c6f33bbdc3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08189ed1a4f44f898d9c1c6f33bbdc3d" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 39677 } }
I20260926 06:19:00.225102 20063 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08189ed1a4f44f898d9c1c6f33bbdc3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:19:00.225126 20063 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08189ed1a4f44f898d9c1c6f33bbdc3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:19:00.225157 20063 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08189ed1a4f44f898d9c1c6f33bbdc3d [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:19:00.225778 20063 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08189ed1a4f44f898d9c1c6f33bbdc3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08189ed1a4f44f898d9c1c6f33bbdc3d" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 39677 } }
I20260926 06:19:00.225874 20063 leader_election.cc:302] T 00000000000000000000000000000000 P 08189ed1a4f44f898d9c1c6f33bbdc3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08189ed1a4f44f898d9c1c6f33bbdc3d; no voters:
I20260926 06:19:00.226023 20063 leader_election.cc:288] T 00000000000000000000000000000000 P 08189ed1a4f44f898d9c1c6f33bbdc3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:19:00.226114 20066 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08189ed1a4f44f898d9c1c6f33bbdc3d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:19:00.226306 20063 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08189ed1a4f44f898d9c1c6f33bbdc3d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:19:00.226342 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 19996
I20260926 06:19:00.226317 20066 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08189ed1a4f44f898d9c1c6f33bbdc3d [term 1 LEADER]: Becoming Leader. State: Replica: 08189ed1a4f44f898d9c1c6f33bbdc3d, State: Running, Role: LEADER
I20260926 06:19:00.226436 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/master-0/wal/instance
I20260926 06:19:00.226563 20066 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08189ed1a4f44f898d9c1c6f33bbdc3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08189ed1a4f44f898d9c1c6f33bbdc3d" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 39677 } }
I20260926 06:19:00.226639 2491 external_mini_cluster.cc:1428] Setting key afa87695fb8961401b9915cb9a3b5aca
I20260926 06:19:00.226965 20066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08189ed1a4f44f898d9c1c6f33bbdc3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08189ed1a4f44f898d9c1c6f33bbdc3d. Latest consensus state: current_term: 1 leader_uuid: "08189ed1a4f44f898d9c1c6f33bbdc3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08189ed1a4f44f898d9c1c6f33bbdc3d" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 39677 } } }
I20260926 06:19:00.227054 20066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08189ed1a4f44f898d9c1c6f33bbdc3d [sys.catalog]: This master's current role is: LEADER
I20260926 06:19:00.227104 20067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08189ed1a4f44f898d9c1c6f33bbdc3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08189ed1a4f44f898d9c1c6f33bbdc3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08189ed1a4f44f898d9c1c6f33bbdc3d" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 39677 } } }
I20260926 06:19:00.227166 20067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08189ed1a4f44f898d9c1c6f33bbdc3d [sys.catalog]: This master's current role is: LEADER
I20260926 06:19:00.227281 20076 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:19:00.227828 20076 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:19:00.229763 20076 catalog_manager.cc:1347] Generated new cluster ID: addbdb2296604add81026e0f4170a209
I20260926 06:19:00.229827 20076 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:19:00.233093 20077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:00.227269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56342 (local address 127.2.110.254:39677)
0926 06:19:00.227497 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:00.227507 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:00.227782 (+ 275us) server_negotiation.cc:407] Connection header received
0926 06:19:00.227920 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:00.227928 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:00.227994 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:00.228148 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:00.228499 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:00.230014 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:00.230922 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:00.231396 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:00.231540 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:00.232340 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:00.232350 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:00.232350 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:19:00.232552 (+ 202us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:00.232585 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:19:00.232597 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:00.232683 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:00.232765 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 06:19:00.232852 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":77,"threads_started":1}
I20260926 06:19:00.247632 20076 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:19:00.248301 20076 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:19:00.251941 20076 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08189ed1a4f44f898d9c1c6f33bbdc3d: Generated new TSK 0
I20260926 06:19:00.252133 20076 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:19:00.257803 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.193:0
--local_ip_for_outbound_sockets=127.2.110.193
--webserver_interface=127.2.110.193
--webserver_port=0
--tserver_master_addrs=127.2.110.254:39677
--builtin_ntp_servers=127.2.110.212:37611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 06:19:00.363256 20087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:19:00.363459 20087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:19:00.363492 20087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:19:00.364706 20087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:19:00.364755 20087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:19:00.364784 20087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.193
I20260926 06:19:00.366253 20087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:37611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.110.193
--webserver_port=0
--tserver_master_addrs=127.2.110.254:39677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.193
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:19:00.366520 20087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:19:00.366851 20087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:19:00.372309 20092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:00.372280 20093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:00.372550 20087 server_base.cc:1034] running on GCE node
W20260926 06:19:00.372331 20095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:00.372852 20087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:19:00.373126 20087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:19:00.374341 20087 hybrid_clock.cc:648] HybridClock initialized: now 1790403540374313 us; error 53 us; skew 500 ppm
I20260926 06:19:00.375749 20087 webserver.cc:460] Webserver started at http://127.2.110.193:44541/ using document root <none> and password file <none>
I20260926 06:19:00.376030 20087 fs_manager.cc:359] Metadata directory not provided
I20260926 06:19:00.376101 20087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:19:00.376252 20087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:19:00.377398 20087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/ts-0/data/instance:
uuid: "703722d9cfe64fcfa401c37fbb1adbe0"
format_stamp: "Formatted at 2026-09-26 06:19:00 on dist-test-slave-hndd"
server_key: "87a1c8dfb9361feb2117e5185a1bfba4"
server_key_iv: "cce50a8105fa79f4d9d729080dd4e382"
server_key_version: "encryptionkey@0"
I20260926 06:19:00.377818 20087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/ts-0/wal/instance:
uuid: "703722d9cfe64fcfa401c37fbb1adbe0"
format_stamp: "Formatted at 2026-09-26 06:19:00 on dist-test-slave-hndd"
server_key: "87a1c8dfb9361feb2117e5185a1bfba4"
server_key_iv: "cce50a8105fa79f4d9d729080dd4e382"
server_key_version: "encryptionkey@0"
I20260926 06:19:00.379256 20087 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:19:00.380091 20101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:00.380299 20087 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:19:00.380414 20087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/ts-0/wal
uuid: "703722d9cfe64fcfa401c37fbb1adbe0"
format_stamp: "Formatted at 2026-09-26 06:19:00 on dist-test-slave-hndd"
server_key: "87a1c8dfb9361feb2117e5185a1bfba4"
server_key_iv: "cce50a8105fa79f4d9d729080dd4e382"
server_key_version: "encryptionkey@0"
I20260926 06:19:00.380502 20087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:19:00.396377 20087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:19:00.396723 20087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:19:00.396878 20087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:19:00.397130 20087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:19:00.397516 20087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:19:00.397570 20087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:00.397610 20087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:19:00.397639 20087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:00.403731 20087 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.193:34291
I20260926 06:19:00.403780 20214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.193:34291 every 8 connection(s)
I20260926 06:19:00.404155 20087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
I20260926 06:19:00.409744 20077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:00.405135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:52085 (local address 127.2.110.254:39677)
0926 06:19:00.405207 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:00.405209 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:00.405513 (+ 304us) server_negotiation.cc:407] Connection header received
0926 06:19:00.406002 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:00.406003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:00.406019 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:00.406073 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:00.406761 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:00.407252 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:00.408240 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:00.408433 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:00.408486 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:00.409479 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:00.409480 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:00.409481 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:19:00.409486 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:00.409501 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:19:00.409505 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:00.409558 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:00.409648 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 06:19:00.409679 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 06:19:00.409929 20217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:00.405123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:39677 (local address 127.2.110.193:52085)
0926 06:19:00.405311 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:00.405416 (+ 105us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:00.405909 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:00.406141 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:19:00.406144 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:19:00.406152 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:19:00.406684 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:00.406689 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:00.407341 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:00.407342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:00.408160 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:00.408162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:00.408513 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:00.408514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:00.408633 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:00.409268 (+ 635us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:19:00.409268 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:19:00.409297 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:19:00.409322 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:19:00.409572 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:19:00.409578 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:19:00.409580 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 06:19:00.409634 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 06:19:00.409721 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":93,"threads_started":1}
I20260926 06:19:00.410705 20215 heartbeater.cc:347] Connected to a master server at 127.2.110.254:39677
I20260926 06:19:00.410796 20215 heartbeater.cc:464] Registering TS with master...
I20260926 06:19:00.410956 20215 heartbeater.cc:511] Master 127.2.110.254:39677 requested a full tablet report, sending...
I20260926 06:19:00.411504 20027 ts_manager.cc:194] Registered new tserver with Master: 703722d9cfe64fcfa401c37fbb1adbe0 (127.2.110.193:34291)
I20260926 06:19:00.412091 20027 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.110.193:52085
I20260926 06:19:00.413859 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 20087
I20260926 06:19:00.413987 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/ts-0/wal/instance
I20260926 06:19:00.414206 2491 external_mini_cluster.cc:1428] Setting key ad8be2f5931c35c10b3dcf327031d18e
I20260926 06:19:00.415102 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.194:0
--local_ip_for_outbound_sockets=127.2.110.194
--webserver_interface=127.2.110.194
--webserver_port=0
--tserver_master_addrs=127.2.110.254:39677
--builtin_ntp_servers=127.2.110.212:37611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 06:19:00.498976 20218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:19:00.499197 20218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:19:00.499220 20218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:19:00.500425 20218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:19:00.500471 20218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:19:00.500499 20218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.194
I20260926 06:19:00.501930 20218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:37611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.110.194
--webserver_port=0
--tserver_master_addrs=127.2.110.254:39677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.194
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:19:00.502187 20218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:19:00.502504 20218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:19:00.507702 20226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:00.507772 20224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:00.507702 20223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:00.507983 20218 server_base.cc:1034] running on GCE node
I20260926 06:19:00.508193 20218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:19:00.508404 20218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:19:00.509595 20218 hybrid_clock.cc:648] HybridClock initialized: now 1790403540509577 us; error 34 us; skew 500 ppm
I20260926 06:19:00.510896 20218 webserver.cc:460] Webserver started at http://127.2.110.194:35715/ using document root <none> and password file <none>
I20260926 06:19:00.511155 20218 fs_manager.cc:359] Metadata directory not provided
I20260926 06:19:00.511255 20218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:19:00.511413 20218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:19:00.512535 20218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/ts-1/data/instance:
uuid: "a190731e62e74b39897cd785f313a844"
format_stamp: "Formatted at 2026-09-26 06:19:00 on dist-test-slave-hndd"
server_key: "dcaab3f10a9f433e8c0b82339e9fcc12"
server_key_iv: "32ec511df3a6a10712b20673d443b4ac"
server_key_version: "encryptionkey@0"
I20260926 06:19:00.512920 20218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/ts-1/wal/instance:
uuid: "a190731e62e74b39897cd785f313a844"
format_stamp: "Formatted at 2026-09-26 06:19:00 on dist-test-slave-hndd"
server_key: "dcaab3f10a9f433e8c0b82339e9fcc12"
server_key_iv: "32ec511df3a6a10712b20673d443b4ac"
server_key_version: "encryptionkey@0"
I20260926 06:19:00.515174 20218 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260926 06:19:00.516264 20232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:00.516474 20218 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:19:00.516574 20218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/ts-1/wal
uuid: "a190731e62e74b39897cd785f313a844"
format_stamp: "Formatted at 2026-09-26 06:19:00 on dist-test-slave-hndd"
server_key: "dcaab3f10a9f433e8c0b82339e9fcc12"
server_key_iv: "32ec511df3a6a10712b20673d443b4ac"
server_key_version: "encryptionkey@0"
I20260926 06:19:00.516664 20218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:19:00.536473 20218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:19:00.536834 20218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:19:00.536994 20218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:19:00.537240 20218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:19:00.537626 20218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:19:00.537684 20218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:00.537729 20218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:19:00.537757 20218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:00.544102 20218 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.194:44115
I20260926 06:19:00.544165 20345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.194:44115 every 8 connection(s)
I20260926 06:19:00.544538 20218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
I20260926 06:19:00.550294 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 20218
I20260926 06:19:00.550429 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/ts-1/wal/instance
I20260926 06:19:00.550631 2491 external_mini_cluster.cc:1428] Setting key f68099db20b56914a621a819b4b5e638
I20260926 06:19:00.550655 20077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:00.545759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:52723 (local address 127.2.110.254:39677)
0926 06:19:00.545838 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:00.545839 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:00.546183 (+ 344us) server_negotiation.cc:407] Connection header received
0926 06:19:00.546683 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:00.546684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:00.546694 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:00.546760 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:00.547501 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:00.548005 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:00.549009 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:00.549194 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:00.549272 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:00.550280 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:00.550281 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:00.550281 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:19:00.550286 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:00.550302 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:19:00.550307 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:00.550428 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:00.550545 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 06:19:00.550581 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 06:19:00.550948 20348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:00.545785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:39677 (local address 127.2.110.194:52723)
0926 06:19:00.545994 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:00.546033 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:00.546547 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:00.546844 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:19:00.546847 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:19:00.546855 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:19:00.547390 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:00.547395 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:00.548110 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:00.548111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:00.548916 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:00.548919 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:00.549328 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:00.549330 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:00.549470 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:00.550018 (+ 548us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:19:00.550019 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:19:00.550092 (+ 73us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:19:00.550124 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:19:00.550435 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:19:00.550441 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:19:00.550445 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:19:00.550526 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 06:19:00.550678 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":97,"threads_started":1}
I20260926 06:19:00.551517 20346 heartbeater.cc:347] Connected to a master server at 127.2.110.254:39677
I20260926 06:19:00.551615 20346 heartbeater.cc:464] Registering TS with master...
I20260926 06:19:00.551775 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.195:0
--local_ip_for_outbound_sockets=127.2.110.195
--webserver_interface=127.2.110.195
--webserver_port=0
--tserver_master_addrs=127.2.110.254:39677
--builtin_ntp_servers=127.2.110.212:37611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 06:19:00.551882 20346 heartbeater.cc:511] Master 127.2.110.254:39677 requested a full tablet report, sending...
I20260926 06:19:00.552490 20026 ts_manager.cc:194] Registered new tserver with Master: a190731e62e74b39897cd785f313a844 (127.2.110.194:44115)
I20260926 06:19:00.553395 20026 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.110.194:52723
W20260926 06:19:00.634045 20349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:19:00.634310 20349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:19:00.634349 20349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:19:00.635528 20349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:19:00.635600 20349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:19:00.635649 20349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.195
I20260926 06:19:00.637056 20349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:37611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.110.195
--webserver_port=0
--tserver_master_addrs=127.2.110.254:39677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.195
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:19:00.637379 20349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:19:00.637768 20349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:19:00.644492 20355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:00.644464 20354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:00.644454 20357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:00.646035 20349 server_base.cc:1034] running on GCE node
I20260926 06:19:00.646241 20349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:19:00.646468 20349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:19:00.647634 20349 hybrid_clock.cc:648] HybridClock initialized: now 1790403540647621 us; error 33 us; skew 500 ppm
I20260926 06:19:00.648999 20349 webserver.cc:460] Webserver started at http://127.2.110.195:46647/ using document root <none> and password file <none>
I20260926 06:19:00.649272 20349 fs_manager.cc:359] Metadata directory not provided
I20260926 06:19:00.649377 20349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:19:00.649552 20349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:19:00.650668 20349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/ts-2/data/instance:
uuid: "fdb28231039842758a7d4b7eec5c1be7"
format_stamp: "Formatted at 2026-09-26 06:19:00 on dist-test-slave-hndd"
server_key: "11bf5c0784871ce17083807b43b42d15"
server_key_iv: "b40ff89462995fa37ff92ddea6843770"
server_key_version: "encryptionkey@0"
I20260926 06:19:00.651026 20349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/ts-2/wal/instance:
uuid: "fdb28231039842758a7d4b7eec5c1be7"
format_stamp: "Formatted at 2026-09-26 06:19:00 on dist-test-slave-hndd"
server_key: "11bf5c0784871ce17083807b43b42d15"
server_key_iv: "b40ff89462995fa37ff92ddea6843770"
server_key_version: "encryptionkey@0"
I20260926 06:19:00.652364 20349 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:19:00.653158 20363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:00.653391 20349 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:19:00.653522 20349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/ts-2/wal
uuid: "fdb28231039842758a7d4b7eec5c1be7"
format_stamp: "Formatted at 2026-09-26 06:19:00 on dist-test-slave-hndd"
server_key: "11bf5c0784871ce17083807b43b42d15"
server_key_iv: "b40ff89462995fa37ff92ddea6843770"
server_key_version: "encryptionkey@0"
I20260926 06:19:00.653599 20349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:19:00.675880 20349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:19:00.676229 20349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:19:00.676393 20349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:19:00.676630 20349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:19:00.676962 20349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:19:00.677017 20349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:00.677060 20349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:19:00.677085 20349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:00.683063 20349 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.195:46683
I20260926 06:19:00.683107 20476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.195:46683 every 8 connection(s)
I20260926 06:19:00.683447 20349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
I20260926 06:19:00.688133 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 20349
I20260926 06:19:00.688251 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/ts-2/wal/instance
I20260926 06:19:00.688444 2491 external_mini_cluster.cc:1428] Setting key 3b95762daead36cb5aa9aa51699e073f
I20260926 06:19:00.689023 20077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:00.684194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:56699 (local address 127.2.110.254:39677)
0926 06:19:00.684247 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:00.684249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:00.684537 (+ 288us) server_negotiation.cc:407] Connection header received
0926 06:19:00.685084 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:00.685086 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:00.685098 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:00.685153 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:00.685866 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:00.686370 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:00.687359 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:00.687562 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:00.687648 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:00.688668 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:00.688678 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:00.688679 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:19:00.688684 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:00.688704 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:19:00.688709 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:00.688763 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:00.688894 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 06:19:00.688929 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 06:19:00.689242 20479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:00.684194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:39677 (local address 127.2.110.195:56699)
0926 06:19:00.684405 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:00.684442 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:00.684973 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:00.685232 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:19:00.685236 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:19:00.685244 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:19:00.685769 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:00.685773 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:00.686467 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:00.686468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:00.687265 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:00.687267 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:00.687658 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:00.687659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:00.687790 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:00.688462 (+ 672us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:19:00.688463 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:19:00.688492 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:19:00.688539 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:19:00.688779 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:19:00.688786 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:19:00.688789 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:19:00.688882 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 06:19:00.688996 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":90,"threads_started":1}
I20260926 06:19:00.689760 20477 heartbeater.cc:347] Connected to a master server at 127.2.110.254:39677
I20260926 06:19:00.689875 20477 heartbeater.cc:464] Registering TS with master...
I20260926 06:19:00.690058 20477 heartbeater.cc:511] Master 127.2.110.254:39677 requested a full tablet report, sending...
I20260926 06:19:00.690418 20026 ts_manager.cc:194] Registered new tserver with Master: fdb28231039842758a7d4b7eec5c1be7 (127.2.110.195:46683)
I20260926 06:19:00.690773 20026 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.110.195:56699
I20260926 06:19:00.700568 2491 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 06:19:00.705476 20077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:00.702121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56352 (local address 127.2.110.254:39677)
0926 06:19:00.702173 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:00.702175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:00.702328 (+ 153us) server_negotiation.cc:407] Connection header received
0926 06:19:00.702439 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:00.702440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:00.702455 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:00.702501 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:00.702662 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:00.703167 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:00.704056 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:00.704301 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:00.704360 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:00.705127 (+ 767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:00.705128 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:00.705129 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:19:00.705132 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:00.705145 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:19:00.705148 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:00.705193 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:00.705309 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 06:19:00.705402 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 06:19:00.707239 20026 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56352:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:19:00.708745 20026 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:19:00.715492 20485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:00.711146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.193:34291 (local address 127.0.0.1:40612)
0926 06:19:00.711370 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:00.711383 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:00.711446 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:00.711856 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:19:00.711857 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:19:00.711860 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:19:00.711923 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:00.711925 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:00.713013 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:00.713015 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:00.714311 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:00.714314 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:00.715175 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:00.715176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:00.715256 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:00.715270 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 06:19:00.715345 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 06:19:00.715399 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":74,"threads_started":1}
I20260926 06:19:00.715497 20488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:00.711262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40612 (local address 127.2.110.193:34291)
0926 06:19:00.711661 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:00.711666 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:00.711677 (+ 11us) server_negotiation.cc:407] Connection header received
0926 06:19:00.711718 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:00.711720 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:00.711736 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:00.711987 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:19:00.712029 (+ 42us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:00.712893 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:00.714410 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:00.715070 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:00.715171 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:00.715267 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:00.715360 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 06:19:00.715399 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":77,"threads_started":1}
I20260926 06:19:00.715723 20489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:00.711321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.195:46683 (local address 127.0.0.1:33400)
0926 06:19:00.711790 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:00.711801 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:00.711849 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:00.712194 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:19:00.712194 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:19:00.712196 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:19:00.712231 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:00.712232 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:00.713667 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:00.713668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:00.714587 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:00.714590 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:00.715497 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:00.715498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:00.715551 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:00.715554 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:19:00.715584 (+ 30us) client_negotiation.cc:240] Negotiation successful
0926 06:19:00.715627 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":84,"threads_started":1}
I20260926 06:19:00.715852 20487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:00.711166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48324 (local address 127.2.110.194:44115)
0926 06:19:00.711455 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:00.711461 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:00.711473 (+ 12us) server_negotiation.cc:407] Connection header received
0926 06:19:00.711521 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:00.711524 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:00.711542 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:00.711624 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:19:00.711900 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:00.713222 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:00.714481 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:00.715190 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:00.715265 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:00.715360 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:00.715718 (+ 358us) server_negotiation.cc:299] Negotiation successful
0926 06:19:00.715767 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":103,"threads_started":1}
I20260926 06:19:00.716012 20486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:00.711082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.194:44115 (local address 127.0.0.1:48324)
0926 06:19:00.711276 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:00.711319 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:00.711388 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:00.711702 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:19:00.711705 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:19:00.711708 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:19:00.711783 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:00.711786 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:00.713318 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:00.713319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:00.714393 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:00.714396 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:00.715400 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:00.715402 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:00.715471 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:00.715476 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 06:19:00.715523 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 06:19:00.715953 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":61,"threads_started":1}
I20260926 06:19:00.716358 20490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:00.711594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33400 (local address 127.2.110.195:46683)
0926 06:19:00.712014 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:00.712019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:00.712028 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:19:00.712066 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:00.712068 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:00.712084 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:00.712172 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:19:00.712295 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:00.713566 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:00.714683 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:00.715405 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:00.716168 (+ 763us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:00.716238 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:00.716260 (+ 22us) server_negotiation.cc:299] Negotiation successful
0926 06:19:00.716294 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":85,"threads_started":1}
I20260926 06:19:00.716300 20149 tablet_service.cc:1467] Processing CreateTablet for tablet e8916f089dd14c679b2cebdbc36b7455 (DEFAULT_TABLE table=test-table [id=4c1823682cf24e66a871fc03df3c04f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:19:00.716683 20149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8916f089dd14c679b2cebdbc36b7455. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:19:00.718052 20280 tablet_service.cc:1467] Processing CreateTablet for tablet e8916f089dd14c679b2cebdbc36b7455 (DEFAULT_TABLE table=test-table [id=4c1823682cf24e66a871fc03df3c04f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:19:00.718397 20280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8916f089dd14c679b2cebdbc36b7455. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:19:00.718324 20411 tablet_service.cc:1467] Processing CreateTablet for tablet e8916f089dd14c679b2cebdbc36b7455 (DEFAULT_TABLE table=test-table [id=4c1823682cf24e66a871fc03df3c04f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:19:00.718623 20411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8916f089dd14c679b2cebdbc36b7455. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:19:00.720073 20491 tablet_bootstrap.cc:492] T e8916f089dd14c679b2cebdbc36b7455 P 703722d9cfe64fcfa401c37fbb1adbe0: Bootstrap starting.
I20260926 06:19:00.720975 20491 tablet_bootstrap.cc:654] T e8916f089dd14c679b2cebdbc36b7455 P 703722d9cfe64fcfa401c37fbb1adbe0: Neither blocks nor log segments found. Creating new log.
I20260926 06:19:00.721324 20491 log.cc:824] T e8916f089dd14c679b2cebdbc36b7455 P 703722d9cfe64fcfa401c37fbb1adbe0: Log is configured to *not* fsync() on all Append() calls
I20260926 06:19:00.721853 20494 tablet_bootstrap.cc:492] T e8916f089dd14c679b2cebdbc36b7455 P fdb28231039842758a7d4b7eec5c1be7: Bootstrap starting.
I20260926 06:19:00.722088 20492 tablet_bootstrap.cc:492] T e8916f089dd14c679b2cebdbc36b7455 P a190731e62e74b39897cd785f313a844: Bootstrap starting.
I20260926 06:19:00.722194 20491 tablet_bootstrap.cc:492] T e8916f089dd14c679b2cebdbc36b7455 P 703722d9cfe64fcfa401c37fbb1adbe0: No bootstrap required, opened a new log
I20260926 06:19:00.722512 20491 ts_tablet_manager.cc:1397] T e8916f089dd14c679b2cebdbc36b7455 P 703722d9cfe64fcfa401c37fbb1adbe0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 06:19:00.722759 20492 tablet_bootstrap.cc:654] T e8916f089dd14c679b2cebdbc36b7455 P a190731e62e74b39897cd785f313a844: Neither blocks nor log segments found. Creating new log.
I20260926 06:19:00.722865 20494 tablet_bootstrap.cc:654] T e8916f089dd14c679b2cebdbc36b7455 P fdb28231039842758a7d4b7eec5c1be7: Neither blocks nor log segments found. Creating new log.
I20260926 06:19:00.723079 20492 log.cc:824] T e8916f089dd14c679b2cebdbc36b7455 P a190731e62e74b39897cd785f313a844: Log is configured to *not* fsync() on all Append() calls
I20260926 06:19:00.723255 20494 log.cc:824] T e8916f089dd14c679b2cebdbc36b7455 P fdb28231039842758a7d4b7eec5c1be7: Log is configured to *not* fsync() on all Append() calls
I20260926 06:19:00.723814 20492 tablet_bootstrap.cc:492] T e8916f089dd14c679b2cebdbc36b7455 P a190731e62e74b39897cd785f313a844: No bootstrap required, opened a new log
I20260926 06:19:00.723898 20492 ts_tablet_manager.cc:1397] T e8916f089dd14c679b2cebdbc36b7455 P a190731e62e74b39897cd785f313a844: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 06:19:00.724725 20491 raft_consensus.cc:348] T e8916f089dd14c679b2cebdbc36b7455 P 703722d9cfe64fcfa401c37fbb1adbe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703722d9cfe64fcfa401c37fbb1adbe0" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 34291 } } peers { permanent_uuid: "fdb28231039842758a7d4b7eec5c1be7" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 46683 } } peers { permanent_uuid: "a190731e62e74b39897cd785f313a844" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 44115 } }
I20260926 06:19:00.724871 20491 raft_consensus.cc:374] T e8916f089dd14c679b2cebdbc36b7455 P 703722d9cfe64fcfa401c37fbb1adbe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:19:00.724895 20491 raft_consensus.cc:729] T e8916f089dd14c679b2cebdbc36b7455 P 703722d9cfe64fcfa401c37fbb1adbe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 703722d9cfe64fcfa401c37fbb1adbe0, State: Initialized, Role: FOLLOWER
I20260926 06:19:00.725034 20491 consensus_queue.cc:260] T e8916f089dd14c679b2cebdbc36b7455 P 703722d9cfe64fcfa401c37fbb1adbe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703722d9cfe64fcfa401c37fbb1adbe0" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 34291 } } peers { permanent_uuid: "fdb28231039842758a7d4b7eec5c1be7" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 46683 } } peers { permanent_uuid: "a190731e62e74b39897cd785f313a844" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 44115 } }
I20260926 06:19:00.725529 20215 heartbeater.cc:503] Master 127.2.110.254:39677 was elected leader, sending a full tablet report...
I20260926 06:19:00.725602 20491 ts_tablet_manager.cc:1428] T e8916f089dd14c679b2cebdbc36b7455 P 703722d9cfe64fcfa401c37fbb1adbe0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 06:19:00.726197 20494 tablet_bootstrap.cc:492] T e8916f089dd14c679b2cebdbc36b7455 P fdb28231039842758a7d4b7eec5c1be7: No bootstrap required, opened a new log
I20260926 06:19:00.726293 20494 ts_tablet_manager.cc:1397] T e8916f089dd14c679b2cebdbc36b7455 P fdb28231039842758a7d4b7eec5c1be7: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260926 06:19:00.726323 20492 raft_consensus.cc:348] T e8916f089dd14c679b2cebdbc36b7455 P a190731e62e74b39897cd785f313a844 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703722d9cfe64fcfa401c37fbb1adbe0" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 34291 } } peers { permanent_uuid: "fdb28231039842758a7d4b7eec5c1be7" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 46683 } } peers { permanent_uuid: "a190731e62e74b39897cd785f313a844" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 44115 } }
I20260926 06:19:00.726505 20492 raft_consensus.cc:374] T e8916f089dd14c679b2cebdbc36b7455 P a190731e62e74b39897cd785f313a844 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:19:00.726552 20492 raft_consensus.cc:729] T e8916f089dd14c679b2cebdbc36b7455 P a190731e62e74b39897cd785f313a844 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a190731e62e74b39897cd785f313a844, State: Initialized, Role: FOLLOWER
I20260926 06:19:00.726694 20492 consensus_queue.cc:260] T e8916f089dd14c679b2cebdbc36b7455 P a190731e62e74b39897cd785f313a844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703722d9cfe64fcfa401c37fbb1adbe0" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 34291 } } peers { permanent_uuid: "fdb28231039842758a7d4b7eec5c1be7" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 46683 } } peers { permanent_uuid: "a190731e62e74b39897cd785f313a844" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 44115 } }
I20260926 06:19:00.727017 20492 ts_tablet_manager.cc:1428] T e8916f089dd14c679b2cebdbc36b7455 P a190731e62e74b39897cd785f313a844: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 06:19:00.727066 20346 heartbeater.cc:503] Master 127.2.110.254:39677 was elected leader, sending a full tablet report...
I20260926 06:19:00.728879 20494 raft_consensus.cc:348] T e8916f089dd14c679b2cebdbc36b7455 P fdb28231039842758a7d4b7eec5c1be7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703722d9cfe64fcfa401c37fbb1adbe0" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 34291 } } peers { permanent_uuid: "fdb28231039842758a7d4b7eec5c1be7" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 46683 } } peers { permanent_uuid: "a190731e62e74b39897cd785f313a844" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 44115 } }
I20260926 06:19:00.729017 20494 raft_consensus.cc:374] T e8916f089dd14c679b2cebdbc36b7455 P fdb28231039842758a7d4b7eec5c1be7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:19:00.729039 20494 raft_consensus.cc:729] T e8916f089dd14c679b2cebdbc36b7455 P fdb28231039842758a7d4b7eec5c1be7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdb28231039842758a7d4b7eec5c1be7, State: Initialized, Role: FOLLOWER
I20260926 06:19:00.729167 20494 consensus_queue.cc:260] T e8916f089dd14c679b2cebdbc36b7455 P fdb28231039842758a7d4b7eec5c1be7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703722d9cfe64fcfa401c37fbb1adbe0" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 34291 } } peers { permanent_uuid: "fdb28231039842758a7d4b7eec5c1be7" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 46683 } } peers { permanent_uuid: "a190731e62e74b39897cd785f313a844" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 44115 } }
I20260926 06:19:00.729476 20494 ts_tablet_manager.cc:1428] T e8916f089dd14c679b2cebdbc36b7455 P fdb28231039842758a7d4b7eec5c1be7: Time spent starting tablet: real 0.003s user 0.002s sys 0.002s
I20260926 06:19:00.729557 20477 heartbeater.cc:503] Master 127.2.110.254:39677 was elected leader, sending a full tablet report...
W20260926 06:19:00.795274 20347 tablet.cc:2369] T e8916f089dd14c679b2cebdbc36b7455 P a190731e62e74b39897cd785f313a844: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:19:00.871901 20497 raft_consensus.cc:482] T e8916f089dd14c679b2cebdbc36b7455 P 703722d9cfe64fcfa401c37fbb1adbe0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:19:00.872037 20497 raft_consensus.cc:504] T e8916f089dd14c679b2cebdbc36b7455 P 703722d9cfe64fcfa401c37fbb1adbe0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703722d9cfe64fcfa401c37fbb1adbe0" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 34291 } } peers { permanent_uuid: "fdb28231039842758a7d4b7eec5c1be7" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 46683 } } peers { permanent_uuid: "a190731e62e74b39897cd785f313a844" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 44115 } }
I20260926 06:19:00.872385 20497 leader_election.cc:288] T e8916f089dd14c679b2cebdbc36b7455 P 703722d9cfe64fcfa401c37fbb1adbe0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdb28231039842758a7d4b7eec5c1be7 (127.2.110.195:46683), a190731e62e74b39897cd785f313a844 (127.2.110.194:44115)
I20260926 06:19:00.875618 20217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:00.872654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.195:46683 (local address 127.2.110.193:38587)
0926 06:19:00.872721 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:00.872736 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:00.872803 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:00.873024 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:19:00.873025 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:19:00.873027 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:19:00.873108 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:00.873110 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:00.873945 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:00.873946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:00.874720 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:00.874721 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:00.875332 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:00.875334 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:00.875422 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:00.875425 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:19:00.875477 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 06:19:00.875510 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":1}
I20260926 06:19:00.875686 20487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:00.872811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:51327 (local address 127.2.110.194:44115)
0926 06:19:00.872854 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:00.872856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:00.873001 (+ 145us) server_negotiation.cc:407] Connection header received
0926 06:19:00.873032 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:00.873033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:00.873046 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:00.873122 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:19:00.873358 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:00.873812 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:00.874894 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:00.875252 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:00.875345 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:00.875354 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:00.875569 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 06:19:00.875609 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 06:19:00.875648 20490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:00.872746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:38587 (local address 127.2.110.195:46683)
0926 06:19:00.872799 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:00.872802 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:00.872816 (+ 14us) server_negotiation.cc:407] Connection header received
0926 06:19:00.872897 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:00.872898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:00.872909 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:00.872975 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:19:00.873203 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:00.873819 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:00.874825 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:00.875243 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:00.875305 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:00.875349 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:00.875515 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 06:19:00.875564 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 06:19:00.875761 20500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:00.872654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.194:44115 (local address 127.2.110.193:51327)
0926 06:19:00.872868 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:00.872880 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:00.872950 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:00.873139 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:19:00.873139 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:19:00.873141 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:19:00.873254 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:00.873256 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:00.873908 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:00.873909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:00.874784 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:00.874787 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:00.875375 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:00.875376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:00.875456 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:00.875481 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 06:19:00.875546 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 06:19:00.875585 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":35,"thread_start_us":114,"threads_started":1}
I20260926 06:19:00.875995 20431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8916f089dd14c679b2cebdbc36b7455" candidate_uuid: "703722d9cfe64fcfa401c37fbb1adbe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdb28231039842758a7d4b7eec5c1be7" is_pre_election: true
I20260926 06:19:00.876087 20300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8916f089dd14c679b2cebdbc36b7455" candidate_uuid: "703722d9cfe64fcfa401c37fbb1adbe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a190731e62e74b39897cd785f313a844" is_pre_election: true
I20260926 06:19:00.876163 20431 raft_consensus.cc:2454] T e8916f089dd14c679b2cebdbc36b7455 P fdb28231039842758a7d4b7eec5c1be7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 703722d9cfe64fcfa401c37fbb1adbe0 in term 0.
I20260926 06:19:00.876260 20300 raft_consensus.cc:2454] T e8916f089dd14c679b2cebdbc36b7455 P a190731e62e74b39897cd785f313a844 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 703722d9cfe64fcfa401c37fbb1adbe0 in term 0.
I20260926 06:19:00.876463 20105 leader_election.cc:302] T e8916f089dd14c679b2cebdbc36b7455 P 703722d9cfe64fcfa401c37fbb1adbe0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 703722d9cfe64fcfa401c37fbb1adbe0, fdb28231039842758a7d4b7eec5c1be7; no voters:
I20260926 06:19:00.876631 20497 raft_consensus.cc:2781] T e8916f089dd14c679b2cebdbc36b7455 P 703722d9cfe64fcfa401c37fbb1adbe0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:19:00.876703 20497 raft_consensus.cc:482] T e8916f089dd14c679b2cebdbc36b7455 P 703722d9cfe64fcfa401c37fbb1adbe0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:19:00.876722 20497 raft_consensus.cc:3037] T e8916f089dd14c679b2cebdbc36b7455 P 703722d9cfe64fcfa401c37fbb1adbe0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:19:00.877559 20497 raft_consensus.cc:504] T e8916f089dd14c679b2cebdbc36b7455 P 703722d9cfe64fcfa401c37fbb1adbe0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703722d9cfe64fcfa401c37fbb1adbe0" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 34291 } } peers { permanent_uuid: "fdb28231039842758a7d4b7eec5c1be7" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 46683 } } peers { permanent_uuid: "a190731e62e74b39897cd785f313a844" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 44115 } }
I20260926 06:19:00.877727 20497 leader_election.cc:288] T e8916f089dd14c679b2cebdbc36b7455 P 703722d9cfe64fcfa401c37fbb1adbe0 [CANDIDATE]: Term 1 election: Requested vote from peers fdb28231039842758a7d4b7eec5c1be7 (127.2.110.195:46683), a190731e62e74b39897cd785f313a844 (127.2.110.194:44115)
I20260926 06:19:00.877902 20300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8916f089dd14c679b2cebdbc36b7455" candidate_uuid: "703722d9cfe64fcfa401c37fbb1adbe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a190731e62e74b39897cd785f313a844"
I20260926 06:19:00.878000 20300 raft_consensus.cc:3037] T e8916f089dd14c679b2cebdbc36b7455 P a190731e62e74b39897cd785f313a844 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:19:00.878758 20300 raft_consensus.cc:2454] T e8916f089dd14c679b2cebdbc36b7455 P a190731e62e74b39897cd785f313a844 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 703722d9cfe64fcfa401c37fbb1adbe0 in term 1.
I20260926 06:19:00.878911 20431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8916f089dd14c679b2cebdbc36b7455" candidate_uuid: "703722d9cfe64fcfa401c37fbb1adbe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdb28231039842758a7d4b7eec5c1be7"
I20260926 06:19:00.878974 20104 leader_election.cc:302] T e8916f089dd14c679b2cebdbc36b7455 P 703722d9cfe64fcfa401c37fbb1adbe0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 703722d9cfe64fcfa401c37fbb1adbe0, a190731e62e74b39897cd785f313a844; no voters:
I20260926 06:19:00.879055 20497 raft_consensus.cc:2781] T e8916f089dd14c679b2cebdbc36b7455 P 703722d9cfe64fcfa401c37fbb1adbe0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:19:00.879017 20431 raft_consensus.cc:3037] T e8916f089dd14c679b2cebdbc36b7455 P fdb28231039842758a7d4b7eec5c1be7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:19:00.879221 20497 raft_consensus.cc:686] T e8916f089dd14c679b2cebdbc36b7455 P 703722d9cfe64fcfa401c37fbb1adbe0 [term 1 LEADER]: Becoming Leader. State: Replica: 703722d9cfe64fcfa401c37fbb1adbe0, State: Running, Role: LEADER
I20260926 06:19:00.879341 20497 consensus_queue.cc:237] T e8916f089dd14c679b2cebdbc36b7455 P 703722d9cfe64fcfa401c37fbb1adbe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703722d9cfe64fcfa401c37fbb1adbe0" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 34291 } } peers { permanent_uuid: "fdb28231039842758a7d4b7eec5c1be7" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 46683 } } peers { permanent_uuid: "a190731e62e74b39897cd785f313a844" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 44115 } }
I20260926 06:19:00.879976 20431 raft_consensus.cc:2454] T e8916f089dd14c679b2cebdbc36b7455 P fdb28231039842758a7d4b7eec5c1be7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 703722d9cfe64fcfa401c37fbb1adbe0 in term 1.
I20260926 06:19:00.880229 20026 catalog_manager.cc:5495] T e8916f089dd14c679b2cebdbc36b7455 P 703722d9cfe64fcfa401c37fbb1adbe0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 703722d9cfe64fcfa401c37fbb1adbe0 (127.2.110.193). New cstate: current_term: 1 leader_uuid: "703722d9cfe64fcfa401c37fbb1adbe0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703722d9cfe64fcfa401c37fbb1adbe0" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 34291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdb28231039842758a7d4b7eec5c1be7" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 46683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a190731e62e74b39897cd785f313a844" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 44115 } health_report { overall_health: UNKNOWN } } }
I20260926 06:19:00.902793 20488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:00.900003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40616 (local address 127.2.110.193:34291)
0926 06:19:00.900051 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:00.900053 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:00.900075 (+ 22us) server_negotiation.cc:407] Connection header received
0926 06:19:00.900113 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:00.900114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:00.900135 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:00.900192 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:19:00.900288 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:00.900807 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:00.901749 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:00.901972 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:00.902040 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:00.902380 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:19:00.902501 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:19:00.902581 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:00.902642 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 06:19:00.902663 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260926 06:19:00.904891 20216 tablet.cc:2369] T e8916f089dd14c679b2cebdbc36b7455 P 703722d9cfe64fcfa401c37fbb1adbe0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:19:00.904927 20431 raft_consensus.cc:1261] T e8916f089dd14c679b2cebdbc36b7455 P fdb28231039842758a7d4b7eec5c1be7 [term 1 FOLLOWER]: Refusing update from remote peer 703722d9cfe64fcfa401c37fbb1adbe0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:19:00.904994 20300 raft_consensus.cc:1261] T e8916f089dd14c679b2cebdbc36b7455 P a190731e62e74b39897cd785f313a844 [term 1 FOLLOWER]: Refusing update from remote peer 703722d9cfe64fcfa401c37fbb1adbe0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:19:00.905382 20501 consensus_queue.cc:1035] T e8916f089dd14c679b2cebdbc36b7455 P 703722d9cfe64fcfa401c37fbb1adbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a190731e62e74b39897cd785f313a844" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 44115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:19:00.905531 20497 consensus_queue.cc:1035] T e8916f089dd14c679b2cebdbc36b7455 P 703722d9cfe64fcfa401c37fbb1adbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdb28231039842758a7d4b7eec5c1be7" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 46683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:19:00.907920 20504 mvcc.cc:205] Tried to move back new op lower bound from 7333492903543136256 to 7333492903442325504. Current Snapshot: MvccSnapshot[applied={T|T < 7333492903543136256}]
I20260926 06:19:00.908658 20505 mvcc.cc:205] Tried to move back new op lower bound from 7333492903543136256 to 7333492903442325504. Current Snapshot: MvccSnapshot[applied={T|T < 7333492903543136256}]
I20260926 06:19:00.910326 20026 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:19:00.910431 20026 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:19:00.911209 20026 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 08189ed1a4f44f898d9c1c6f33bbdc3d: Sending DeleteTablet for 3 replicas of tablet e8916f089dd14c679b2cebdbc36b7455
I20260926 06:19:00.911592 20411 tablet_service.cc:1514] Processing DeleteTablet for tablet e8916f089dd14c679b2cebdbc36b7455 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:19:00 UTC) from {username='slave'} at 127.0.0.1:33400
I20260926 06:19:00.911602 20149 tablet_service.cc:1514] Processing DeleteTablet for tablet e8916f089dd14c679b2cebdbc36b7455 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:19:00 UTC) from {username='slave'} at 127.0.0.1:40612
I20260926 06:19:00.911886 20280 tablet_service.cc:1514] Processing DeleteTablet for tablet e8916f089dd14c679b2cebdbc36b7455 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:19:00 UTC) from {username='slave'} at 127.0.0.1:48324
I20260926 06:19:00.912175 20516 tablet_replica.cc:330] T e8916f089dd14c679b2cebdbc36b7455 P a190731e62e74b39897cd785f313a844: stopping tablet replica
I20260926 06:19:00.912221 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 20087
I20260926 06:19:00.912307 20516 raft_consensus.cc:2229] T e8916f089dd14c679b2cebdbc36b7455 P a190731e62e74b39897cd785f313a844 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:19:00.912763 20516 raft_consensus.cc:2258] T e8916f089dd14c679b2cebdbc36b7455 P a190731e62e74b39897cd785f313a844 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:19:00.912820 20514 tablet_replica.cc:330] T e8916f089dd14c679b2cebdbc36b7455 P fdb28231039842758a7d4b7eec5c1be7: stopping tablet replica
I20260926 06:19:00.912919 20514 raft_consensus.cc:2229] T e8916f089dd14c679b2cebdbc36b7455 P fdb28231039842758a7d4b7eec5c1be7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:19:00.913170 20514 raft_consensus.cc:2258] T e8916f089dd14c679b2cebdbc36b7455 P fdb28231039842758a7d4b7eec5c1be7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:19:00.913280 20516 ts_tablet_manager.cc:1905] T e8916f089dd14c679b2cebdbc36b7455 P a190731e62e74b39897cd785f313a844: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:19:00.915350 20516 ts_tablet_manager.cc:1918] T e8916f089dd14c679b2cebdbc36b7455 P a190731e62e74b39897cd785f313a844: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:19:00.915468 20516 log.cc:1196] T e8916f089dd14c679b2cebdbc36b7455 P a190731e62e74b39897cd785f313a844: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/ts-1/wal/wals/e8916f089dd14c679b2cebdbc36b7455
I20260926 06:19:00.915737 20516 ts_tablet_manager.cc:1939] T e8916f089dd14c679b2cebdbc36b7455 P a190731e62e74b39897cd785f313a844: Deleting consensus metadata
I20260926 06:19:00.916218 20013 catalog_manager.cc:4841] TS a190731e62e74b39897cd785f313a844 (127.2.110.194:44115): tablet e8916f089dd14c679b2cebdbc36b7455 (table test-table [id=4c1823682cf24e66a871fc03df3c04f1]) successfully deleted
I20260926 06:19:00.917673 20514 ts_tablet_manager.cc:1905] T e8916f089dd14c679b2cebdbc36b7455 P fdb28231039842758a7d4b7eec5c1be7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 06:19:00.917982 20014 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.110.193:34291 (error 108)
I20260926 06:19:00.918565 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 20218
I20260926 06:19:00.918584 20485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:00.918425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.193:34291 (local address 127.0.0.1:40628)
0926 06:19:00.918501 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:00.918549 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.193:34291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 06:19:00.919399 20014 catalog_manager.cc:4584] TS 703722d9cfe64fcfa401c37fbb1adbe0 (127.2.110.193:34291): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e8916f089dd14c679b2cebdbc36b7455: Network error: Client connection negotiation failed: client connection to 127.2.110.193:34291: connect: Connection refused (error 111)
I20260926 06:19:00.920394 20514 ts_tablet_manager.cc:1918] T e8916f089dd14c679b2cebdbc36b7455 P fdb28231039842758a7d4b7eec5c1be7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:19:00.920492 20514 log.cc:1196] T e8916f089dd14c679b2cebdbc36b7455 P fdb28231039842758a7d4b7eec5c1be7: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403292369990-2491-0/minicluster-data/ts-2/wal/wals/e8916f089dd14c679b2cebdbc36b7455
I20260926 06:19:00.920748 20514 ts_tablet_manager.cc:1939] T e8916f089dd14c679b2cebdbc36b7455 P fdb28231039842758a7d4b7eec5c1be7: Deleting consensus metadata
I20260926 06:19:00.921200 20014 catalog_manager.cc:4841] TS fdb28231039842758a7d4b7eec5c1be7 (127.2.110.195:46683): tablet e8916f089dd14c679b2cebdbc36b7455 (table test-table [id=4c1823682cf24e66a871fc03df3c04f1]) successfully deleted
I20260926 06:19:00.923995 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 20349
I20260926 06:19:00.929407 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 19996
2026-09-26T06:19:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (867 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:19:00 dist-test-slave-hndd krb5kdc[20520](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:19:00 dist-test-slave-hndd krb5kdc[20520](info): set up 2 sockets
Sep 26 06:19:00 dist-test-slave-hndd krb5kdc[20520](info): commencing operation
krb5kdc: starting...
W20260926 06:19:01.989053 2491 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Sep 26 06:19:02 dist-test-slave-hndd krb5kdc[20520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:19:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:19:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
I20260926 06:19:02.114534 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:33997
--webserver_interface=127.2.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:41047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.110.254:33997
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:19:02.199544 20533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:19:02.199838 20533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:19:02.199885 20533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:19:02.201203 20533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:19:02.201290 20533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:19:02.201329 20533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:19:02.201419 20533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:19:02.201452 20533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:19:02.203191 20533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:41047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.110.254:33997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:33997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:19:02.203523 20533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:19:02.203791 20533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:19:02.209576 20541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:02.209735 20533 server_base.cc:1034] running on GCE node
W20260926 06:19:02.209550 20539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:02.209837 20538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:02.210201 20533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:19:02.210471 20533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:19:02.211627 20533 hybrid_clock.cc:648] HybridClock initialized: now 1790403542211613 us; error 30 us; skew 500 ppm
Sep 26 06:19:02 dist-test-slave-hndd krb5kdc[20520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403542, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:19:02.214376 20533 init.cc:377] Logged in from keytab as kudu/127.2.110.254@KRBTEST.COM (short username kudu)
I20260926 06:19:02.214885 20533 webserver.cc:460] Webserver started at http://127.2.110.254:44663/ using document root <none> and password file <none>
I20260926 06:19:02.215142 20533 fs_manager.cc:359] Metadata directory not provided
I20260926 06:19:02.215214 20533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:19:02.215361 20533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:19:02.216423 20533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/master-0/data/instance:
uuid: "cf816272cb064d8499deb48ac222d5f7"
format_stamp: "Formatted at 2026-09-26 06:19:02 on dist-test-slave-hndd"
server_key: "822fd66866ea7e4a6efc434eecd81f2f"
server_key_iv: "ef5550872ea9be06562b05417cf4e7ff"
server_key_version: "encryptionkey@0"
I20260926 06:19:02.216780 20533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/master-0/wal/instance:
uuid: "cf816272cb064d8499deb48ac222d5f7"
format_stamp: "Formatted at 2026-09-26 06:19:02 on dist-test-slave-hndd"
server_key: "822fd66866ea7e4a6efc434eecd81f2f"
server_key_iv: "ef5550872ea9be06562b05417cf4e7ff"
server_key_version: "encryptionkey@0"
I20260926 06:19:02.218303 20533 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:19:02.219048 20548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:02.219265 20533 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:19:02.219369 20533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "cf816272cb064d8499deb48ac222d5f7"
format_stamp: "Formatted at 2026-09-26 06:19:02 on dist-test-slave-hndd"
server_key: "822fd66866ea7e4a6efc434eecd81f2f"
server_key_iv: "ef5550872ea9be06562b05417cf4e7ff"
server_key_version: "encryptionkey@0"
I20260926 06:19:02.219457 20533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:19:02.230952 20533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:19:02.233433 20533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:19:02.233608 20533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:19:02.237798 20533 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.254:33997
I20260926 06:19:02.237855 20600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.254:33997 every 8 connection(s)
I20260926 06:19:02.238238 20533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:19:02.239463 20601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:19:02.239796 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 20533
I20260926 06:19:02.239909 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/master-0/wal/instance
I20260926 06:19:02.240118 2491 external_mini_cluster.cc:1428] Setting key a805fc424cc0546044d66964c6f23505
I20260926 06:19:02.242463 20601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf816272cb064d8499deb48ac222d5f7: Bootstrap starting.
I20260926 06:19:02.243225 20601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf816272cb064d8499deb48ac222d5f7: Neither blocks nor log segments found. Creating new log.
I20260926 06:19:02.243548 20601 log.cc:824] T 00000000000000000000000000000000 P cf816272cb064d8499deb48ac222d5f7: Log is configured to *not* fsync() on all Append() calls
I20260926 06:19:02.244298 20601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf816272cb064d8499deb48ac222d5f7: No bootstrap required, opened a new log
I20260926 06:19:02.246750 20601 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf816272cb064d8499deb48ac222d5f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf816272cb064d8499deb48ac222d5f7" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 33997 } }
I20260926 06:19:02.246907 20601 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf816272cb064d8499deb48ac222d5f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:19:02.246932 20601 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf816272cb064d8499deb48ac222d5f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf816272cb064d8499deb48ac222d5f7, State: Initialized, Role: FOLLOWER
I20260926 06:19:02.247071 20601 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf816272cb064d8499deb48ac222d5f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf816272cb064d8499deb48ac222d5f7" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 33997 } }
I20260926 06:19:02.247157 20601 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf816272cb064d8499deb48ac222d5f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:19:02.247210 20601 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf816272cb064d8499deb48ac222d5f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:19:02.247259 20601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf816272cb064d8499deb48ac222d5f7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:19:02.247920 20601 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf816272cb064d8499deb48ac222d5f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf816272cb064d8499deb48ac222d5f7" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 33997 } }
I20260926 06:19:02.248031 20601 leader_election.cc:302] T 00000000000000000000000000000000 P cf816272cb064d8499deb48ac222d5f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf816272cb064d8499deb48ac222d5f7; no voters:
I20260926 06:19:02.248236 20601 leader_election.cc:288] T 00000000000000000000000000000000 P cf816272cb064d8499deb48ac222d5f7 [CANDIDATE]: Term 1 election: Requested vote from peers
Sep 26 06:19:02 dist-test-slave-hndd krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:19:02.248436 20606 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf816272cb064d8499deb48ac222d5f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:19:02.248517 20601 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf816272cb064d8499deb48ac222d5f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:19:02.248606 20606 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf816272cb064d8499deb48ac222d5f7 [term 1 LEADER]: Becoming Leader. State: Replica: cf816272cb064d8499deb48ac222d5f7, State: Running, Role: LEADER
I20260926 06:19:02.248718 20606 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf816272cb064d8499deb48ac222d5f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf816272cb064d8499deb48ac222d5f7" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 33997 } }
I20260926 06:19:02.249047 20606 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf816272cb064d8499deb48ac222d5f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf816272cb064d8499deb48ac222d5f7. Latest consensus state: current_term: 1 leader_uuid: "cf816272cb064d8499deb48ac222d5f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf816272cb064d8499deb48ac222d5f7" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 33997 } } }
I20260926 06:19:02.249152 20606 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf816272cb064d8499deb48ac222d5f7 [sys.catalog]: This master's current role is: LEADER
I20260926 06:19:02.249059 20607 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf816272cb064d8499deb48ac222d5f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf816272cb064d8499deb48ac222d5f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf816272cb064d8499deb48ac222d5f7" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 33997 } } }
I20260926 06:19:02.249269 20607 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf816272cb064d8499deb48ac222d5f7 [sys.catalog]: This master's current role is: LEADER
W20260926 06:19:02.251196 20621 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cf816272cb064d8499deb48ac222d5f7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 06:19:02.251314 20621 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 06:19:02.251407 20614 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:19:02.251564 20614 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:19:02.252853 20604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:02.240881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37506 (local address 127.2.110.254:33997)
0926 06:19:02.241201 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:02.241212 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:02.241257 (+ 45us) server_negotiation.cc:407] Connection header received
0926 06:19:02.241774 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:02.241780 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:02.241828 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:02.242021 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:02.242701 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:02.244100 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:02.245071 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:02.245743 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:02.245880 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:02.248965 (+ 3085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:02.248970 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:02.248970 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:19:02.248984 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:02.251164 (+ 2180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:02.251564 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:02.251567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:02.251568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:02.251658 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:02.251883 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:02.251884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:02.251884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:02.252205 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:02.252361 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:02.252529 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 06:19:02.252688 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":89,"threads_started":1}
I20260926 06:19:02.253310 20614 catalog_manager.cc:1347] Generated new cluster ID: 6a132370d99f466f96251bf7f8df5fd6
I20260926 06:19:02.253391 20614 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:19:02.266575 20614 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:19:02.267181 20614 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:19:02.277750 20614 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf816272cb064d8499deb48ac222d5f7: Generated new TSK 0
I20260926 06:19:02.277987 20614 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab.
Entry for principal kudu/127.2.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab.
I20260926 06:19:02.317077 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.193:0
--local_ip_for_outbound_sockets=127.2.110.193
--webserver_interface=127.2.110.193
--webserver_port=0
--tserver_master_addrs=127.2.110.254:33997
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab
--principal=kudu/127.2.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:41047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:19:02.427817 20627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:19:02.428043 20627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:19:02.428069 20627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:19:02.429177 20627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:19:02.429219 20627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:19:02.429245 20627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.193
I20260926 06:19:02.430774 20627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:41047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab
--principal=kudu/127.2.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.110.193
--webserver_port=0
--tserver_master_addrs=127.2.110.254:33997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.193
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:19:02.430987 20627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:19:02.431167 20627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:19:02.436687 20632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:02.436687 20635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:02.436689 20633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:02.436966 20627 server_base.cc:1034] running on GCE node
I20260926 06:19:02.437449 20627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:19:02.437759 20627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:19:02.438941 20627 hybrid_clock.cc:648] HybridClock initialized: now 1790403542438921 us; error 49 us; skew 500 ppm
Sep 26 06:19:02 dist-test-slave-hndd krb5kdc[20520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403542, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:19:02.441713 20627 init.cc:377] Logged in from keytab as kudu/127.2.110.193@KRBTEST.COM (short username kudu)
I20260926 06:19:02.442301 20627 webserver.cc:460] Webserver started at http://127.2.110.193:42219/ using document root <none> and password file <none>
I20260926 06:19:02.442584 20627 fs_manager.cc:359] Metadata directory not provided
I20260926 06:19:02.442638 20627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:19:02.442782 20627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:19:02.443895 20627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/ts-0/data/instance:
uuid: "6e9cff8d0011404a9cdfddb007b6db95"
format_stamp: "Formatted at 2026-09-26 06:19:02 on dist-test-slave-hndd"
server_key: "794ebee94b7e56719a4babe18f0bbb4c"
server_key_iv: "5ed879d47e56a16b775e8b98d93f97ed"
server_key_version: "encryptionkey@0"
I20260926 06:19:02.444267 20627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/ts-0/wal/instance:
uuid: "6e9cff8d0011404a9cdfddb007b6db95"
format_stamp: "Formatted at 2026-09-26 06:19:02 on dist-test-slave-hndd"
server_key: "794ebee94b7e56719a4babe18f0bbb4c"
server_key_iv: "5ed879d47e56a16b775e8b98d93f97ed"
server_key_version: "encryptionkey@0"
I20260926 06:19:02.445724 20627 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:19:02.446678 20642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:02.447016 20627 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:19:02.447125 20627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/ts-0/wal
uuid: "6e9cff8d0011404a9cdfddb007b6db95"
format_stamp: "Formatted at 2026-09-26 06:19:02 on dist-test-slave-hndd"
server_key: "794ebee94b7e56719a4babe18f0bbb4c"
server_key_iv: "5ed879d47e56a16b775e8b98d93f97ed"
server_key_version: "encryptionkey@0"
I20260926 06:19:02.447206 20627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:19:02.467236 20627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:19:02.469451 20627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:19:02.469640 20627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:19:02.469888 20627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:19:02.470216 20627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:19:02.470271 20627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:02.470309 20627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:19:02.470338 20627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:02.478073 20627 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.193:36689
I20260926 06:19:02.478120 20755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.193:36689 every 8 connection(s)
I20260926 06:19:02.478479 20627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
I20260926 06:19:02.483697 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 20627
I20260926 06:19:02.483850 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/ts-0/wal/instance
I20260926 06:19:02.484061 2491 external_mini_cluster.cc:1428] Setting key 536494c361547c5bb06181cba5219166
Sep 26 06:19:02 dist-test-slave-hndd krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403542, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.193@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:19:02.489120 20604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:02.479241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:45023 (local address 127.2.110.254:33997)
0926 06:19:02.479325 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:02.479327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:02.479531 (+ 204us) server_negotiation.cc:407] Connection header received
0926 06:19:02.480044 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:02.480045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:02.480056 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:02.480127 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:02.481220 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:02.481866 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:02.482859 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:02.483119 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:02.483189 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:02.485752 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:02.485754 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:02.485754 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:19:02.485762 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:02.487546 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:02.487998 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:02.488000 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:02.488000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:02.488067 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:02.488311 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:02.488312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:02.488312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:02.488478 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:02.488551 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:02.488869 (+ 318us) server_negotiation.cc:299] Negotiation successful
0926 06:19:02.488980 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 06:19:02.489180 20758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:02.479191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:33997 (local address 127.2.110.193:45023)
0926 06:19:02.479370 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:02.479408 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:02.479926 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:02.480282 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:19:02.480287 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:19:02.480901 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:19:02.481100 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:02.481105 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:02.481967 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:02.481967 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:02.482760 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:02.482763 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:02.483205 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:02.483206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:02.483315 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:02.483990 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:19:02.483999 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:19:02.485643 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:19:02.487649 (+ 2006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:19:02.487657 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:19:02.487659 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:19:02.487916 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:19:02.488137 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:19:02.488137 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:19:02.488138 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:19:02.488245 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:19:02.488550 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:19:02.488555 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:19:02.488680 (+ 125us) client_negotiation.cc:769] Sending connection context
0926 06:19:02.488826 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 06:19:02.488990 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":93,"threads_started":1}
I20260926 06:19:02.489737 20756 heartbeater.cc:347] Connected to a master server at 127.2.110.254:33997
I20260926 06:19:02.489848 20756 heartbeater.cc:464] Registering TS with master...
I20260926 06:19:02.490041 20756 heartbeater.cc:511] Master 127.2.110.254:33997 requested a full tablet report, sending...
I20260926 06:19:02.490669 20565 ts_manager.cc:194] Registered new tserver with Master: 6e9cff8d0011404a9cdfddb007b6db95 (127.2.110.193:36689)
I20260926 06:19:02.491665 20565 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.193@KRBTEST.COM'} at 127.2.110.193:45023
WARNING: no policy specified for kudu/127.2.110.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab.
Entry for principal kudu/127.2.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab.
I20260926 06:19:02.517104 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.194:0
--local_ip_for_outbound_sockets=127.2.110.194
--webserver_interface=127.2.110.194
--webserver_port=0
--tserver_master_addrs=127.2.110.254:33997
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab
--principal=kudu/127.2.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:41047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:19:02.601197 20761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:19:02.601575 20761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:19:02.601624 20761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:19:02.602816 20761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:19:02.602906 20761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:19:02.602950 20761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.194
I20260926 06:19:02.604497 20761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:41047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab
--principal=kudu/127.2.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.110.194
--webserver_port=0
--tserver_master_addrs=127.2.110.254:33997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.194
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:19:02.604790 20761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:19:02.605046 20761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:19:02.610201 20767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:02.610190 20769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:02.610222 20766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:02.610399 20761 server_base.cc:1034] running on GCE node
I20260926 06:19:02.610666 20761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:19:02.610926 20761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:19:02.612089 20761 hybrid_clock.cc:648] HybridClock initialized: now 1790403542612079 us; error 32 us; skew 500 ppm
Sep 26 06:19:02 dist-test-slave-hndd krb5kdc[20520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403542, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:19:02.614688 20761 init.cc:377] Logged in from keytab as kudu/127.2.110.194@KRBTEST.COM (short username kudu)
I20260926 06:19:02.615211 20761 webserver.cc:460] Webserver started at http://127.2.110.194:42339/ using document root <none> and password file <none>
I20260926 06:19:02.615509 20761 fs_manager.cc:359] Metadata directory not provided
I20260926 06:19:02.615581 20761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:19:02.615720 20761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:19:02.616801 20761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/ts-1/data/instance:
uuid: "ddac8cd8c066434f81c299fdcc212f7d"
format_stamp: "Formatted at 2026-09-26 06:19:02 on dist-test-slave-hndd"
server_key: "98ebaeac9c3654b497709f03e0727ce9"
server_key_iv: "4b7717eb245efba2ece3f4032d5acf9a"
server_key_version: "encryptionkey@0"
I20260926 06:19:02.617177 20761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/ts-1/wal/instance:
uuid: "ddac8cd8c066434f81c299fdcc212f7d"
format_stamp: "Formatted at 2026-09-26 06:19:02 on dist-test-slave-hndd"
server_key: "98ebaeac9c3654b497709f03e0727ce9"
server_key_iv: "4b7717eb245efba2ece3f4032d5acf9a"
server_key_version: "encryptionkey@0"
I20260926 06:19:02.618600 20761 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:19:02.619510 20776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:02.619740 20761 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 06:19:02.619854 20761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/ts-1/wal
uuid: "ddac8cd8c066434f81c299fdcc212f7d"
format_stamp: "Formatted at 2026-09-26 06:19:02 on dist-test-slave-hndd"
server_key: "98ebaeac9c3654b497709f03e0727ce9"
server_key_iv: "4b7717eb245efba2ece3f4032d5acf9a"
server_key_version: "encryptionkey@0"
I20260926 06:19:02.619948 20761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:19:02.634008 20761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:19:02.636813 20761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:19:02.637034 20761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:19:02.637395 20761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:19:02.637753 20761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:19:02.637814 20761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:02.637857 20761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:19:02.637898 20761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:02.646373 20761 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.194:34907
I20260926 06:19:02.646505 20889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.194:34907 every 8 connection(s)
I20260926 06:19:02.646864 20761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
I20260926 06:19:02.653869 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 20761
I20260926 06:19:02.654045 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/ts-1/wal/instance
I20260926 06:19:02.654350 2491 external_mini_cluster.cc:1428] Setting key b2c18486b61c7e9ebd5ab529ca5856c3
Sep 26 06:19:02 dist-test-slave-hndd krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403542, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.194@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:19:02.658818 20604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:02.647845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:53271 (local address 127.2.110.254:33997)
0926 06:19:02.647898 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:02.647900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:02.648277 (+ 377us) server_negotiation.cc:407] Connection header received
0926 06:19:02.649265 (+ 988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:02.649267 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:02.649283 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:02.649427 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:02.650976 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:02.651559 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:02.652849 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:02.653084 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:02.653157 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:02.655821 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:02.655823 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:02.655823 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:19:02.655832 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:02.657291 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:02.657774 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:02.657775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:02.657776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:02.657839 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:02.658106 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:02.658107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:02.658107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:02.658241 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:02.658315 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:02.658613 (+ 298us) server_negotiation.cc:299] Negotiation successful
0926 06:19:02.658714 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 06:19:02.659040 20892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:02.647833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:33997 (local address 127.2.110.194:53271)
0926 06:19:02.648034 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:02.648082 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:02.649092 (+ 1010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:02.649560 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:19:02.649565 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:19:02.650456 (+ 891us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:19:02.650825 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:02.650844 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:02.651726 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:02.651727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:02.652725 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:02.652728 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:02.653170 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:02.653171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:02.653275 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:02.654190 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:19:02.654199 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:19:02.655713 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:19:02.657426 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:19:02.657435 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:19:02.657438 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:19:02.657689 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:19:02.657915 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:19:02.657916 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:19:02.657916 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:19:02.658036 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:19:02.658317 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:19:02.658325 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:19:02.658478 (+ 153us) client_negotiation.cc:769] Sending connection context
0926 06:19:02.658593 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 06:19:02.658784 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":88,"threads_started":1}
I20260926 06:19:02.659690 20890 heartbeater.cc:347] Connected to a master server at 127.2.110.254:33997
I20260926 06:19:02.659801 20890 heartbeater.cc:464] Registering TS with master...
I20260926 06:19:02.660022 20890 heartbeater.cc:511] Master 127.2.110.254:33997 requested a full tablet report, sending...
I20260926 06:19:02.660660 20565 ts_manager.cc:194] Registered new tserver with Master: ddac8cd8c066434f81c299fdcc212f7d (127.2.110.194:34907)
I20260926 06:19:02.661108 20565 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.194@KRBTEST.COM'} at 127.2.110.194:53271
WARNING: no policy specified for kudu/127.2.110.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab.
Entry for principal kudu/127.2.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab.
I20260926 06:19:02.688104 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.195:0
--local_ip_for_outbound_sockets=127.2.110.195
--webserver_interface=127.2.110.195
--webserver_port=0
--tserver_master_addrs=127.2.110.254:33997
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab
--principal=kudu/127.2.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:41047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:19:02.777405 20895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:19:02.777671 20895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:19:02.777714 20895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:19:02.779071 20895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:19:02.779125 20895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:19:02.779155 20895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.195
I20260926 06:19:02.780611 20895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:41047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab
--principal=kudu/127.2.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.110.195
--webserver_port=0
--tserver_master_addrs=127.2.110.254:33997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.195
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:19:02.780810 20895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:19:02.780997 20895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:19:02.786263 20900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:02.786295 20901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:02.786266 20903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:02.786518 20895 server_base.cc:1034] running on GCE node
I20260926 06:19:02.786744 20895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:19:02.787015 20895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:19:02.788200 20895 hybrid_clock.cc:648] HybridClock initialized: now 1790403542788187 us; error 30 us; skew 500 ppm
Sep 26 06:19:02 dist-test-slave-hndd krb5kdc[20520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403542, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:19:02.790971 20895 init.cc:377] Logged in from keytab as kudu/127.2.110.195@KRBTEST.COM (short username kudu)
I20260926 06:19:02.791448 20895 webserver.cc:460] Webserver started at http://127.2.110.195:42833/ using document root <none> and password file <none>
I20260926 06:19:02.791687 20895 fs_manager.cc:359] Metadata directory not provided
I20260926 06:19:02.791757 20895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:19:02.791924 20895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:19:02.793090 20895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/ts-2/data/instance:
uuid: "264d75b1ad55441eab6a9f1a43f4b218"
format_stamp: "Formatted at 2026-09-26 06:19:02 on dist-test-slave-hndd"
server_key: "35fd61827f369063787d61d00593a730"
server_key_iv: "7bc8a6285a7aa3f800ae6a87a8898fa2"
server_key_version: "encryptionkey@0"
I20260926 06:19:02.793505 20895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/ts-2/wal/instance:
uuid: "264d75b1ad55441eab6a9f1a43f4b218"
format_stamp: "Formatted at 2026-09-26 06:19:02 on dist-test-slave-hndd"
server_key: "35fd61827f369063787d61d00593a730"
server_key_iv: "7bc8a6285a7aa3f800ae6a87a8898fa2"
server_key_version: "encryptionkey@0"
I20260926 06:19:02.794963 20895 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:19:02.795814 20910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:02.796033 20895 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:19:02.796139 20895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/ts-2/wal
uuid: "264d75b1ad55441eab6a9f1a43f4b218"
format_stamp: "Formatted at 2026-09-26 06:19:02 on dist-test-slave-hndd"
server_key: "35fd61827f369063787d61d00593a730"
server_key_iv: "7bc8a6285a7aa3f800ae6a87a8898fa2"
server_key_version: "encryptionkey@0"
I20260926 06:19:02.796226 20895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:19:02.808858 20895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:19:02.811193 20895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:19:02.811352 20895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:19:02.811602 20895 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:19:02.811936 20895 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:19:02.811980 20895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:02.812003 20895 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:19:02.812016 20895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:02.817984 20895 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.195:35011
I20260926 06:19:02.818022 21023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.195:35011 every 8 connection(s)
I20260926 06:19:02.818357 20895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
I20260926 06:19:02.823518 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 20895
I20260926 06:19:02.823604 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/ts-2/wal/instance
I20260926 06:19:02.823797 2491 external_mini_cluster.cc:1428] Setting key 1fd74ba8551cba4952574bfa2fb98d1a
Sep 26 06:19:02 dist-test-slave-hndd krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403542, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.195@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:19:02.828310 20604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:02.819130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:34045 (local address 127.2.110.254:33997)
0926 06:19:02.819213 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:02.819216 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:02.819436 (+ 220us) server_negotiation.cc:407] Connection header received
0926 06:19:02.819941 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:02.819943 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:02.819954 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:02.820014 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:02.820985 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:02.821615 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:02.822710 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:02.822981 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:02.823058 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:02.825717 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:02.825718 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:02.825718 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:19:02.825725 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:02.827065 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:02.827475 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:02.827476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:02.827476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:02.827528 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:02.827750 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:02.827751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:02.827751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:02.827861 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:02.827915 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:02.828166 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 06:19:02.828231 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 06:19:02.828557 21026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:02.819081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:33997 (local address 127.2.110.195:34045)
0926 06:19:02.819303 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:02.819340 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:02.819831 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:02.820093 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:19:02.820096 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:19:02.820688 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:19:02.820884 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:02.820889 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:02.821695 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:02.821696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:02.822589 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:02.822592 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:02.823077 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:02.823078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:02.823180 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:02.823875 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:19:02.823885 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:19:02.825587 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:19:02.827147 (+ 1560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:19:02.827155 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:19:02.827157 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:19:02.827408 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:19:02.827588 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:19:02.827589 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:19:02.827589 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:19:02.827692 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:19:02.827914 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:19:02.827919 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:19:02.828052 (+ 133us) client_negotiation.cc:769] Sending connection context
0926 06:19:02.828138 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 06:19:02.828270 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":108,"threads_started":1}
I20260926 06:19:02.829139 21024 heartbeater.cc:347] Connected to a master server at 127.2.110.254:33997
I20260926 06:19:02.829239 21024 heartbeater.cc:464] Registering TS with master...
I20260926 06:19:02.829473 21024 heartbeater.cc:511] Master 127.2.110.254:33997 requested a full tablet report, sending...
I20260926 06:19:02.829885 20565 ts_manager.cc:194] Registered new tserver with Master: 264d75b1ad55441eab6a9f1a43f4b218 (127.2.110.195:35011)
I20260926 06:19:02.830271 20565 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.195@KRBTEST.COM'} at 127.2.110.195:34045
I20260926 06:19:02.835764 2491 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 06:19:02.843595 20604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:02.837118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37522 (local address 127.2.110.254:33997)
0926 06:19:02.837194 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:02.837197 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:02.837366 (+ 169us) server_negotiation.cc:407] Connection header received
0926 06:19:02.837450 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:02.837451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:02.837466 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:02.837526 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:02.838104 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:02.838591 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:02.839303 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:02.839501 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:02.839576 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:02.840964 (+ 1388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:02.840966 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:02.840966 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:19:02.840975 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:02.842228 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:02.842674 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:02.842676 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:02.842676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:02.842735 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:02.842970 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:02.842971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:02.842971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:02.843125 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:02.843246 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:02.843413 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 06:19:02.843481 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 06:19:02.845916 20565 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37522:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:19:02.847584 20565 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:19:02.854564 21036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:02.850188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.193:36689 (local address 127.0.0.1:35032)
0926 06:19:02.850491 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:02.850503 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:02.850557 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:02.851138 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:19:02.851139 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:19:02.851141 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:19:02.851186 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:02.851187 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:02.852165 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:02.852166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:02.853504 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:02.853507 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:02.854288 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:02.854290 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:02.854367 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:02.854372 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 06:19:02.854429 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 06:19:02.854478 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":69,"thread_start_us":85,"threads_started":1}
I20260926 06:19:02.854565 21037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:02.850297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35032 (local address 127.2.110.193:36689)
0926 06:19:02.850758 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:02.850763 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:02.850915 (+ 152us) server_negotiation.cc:407] Connection header received
0926 06:19:02.850964 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:02.850966 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:02.850982 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:02.851058 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:19:02.851262 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:02.852058 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:02.853598 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:02.854198 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:02.854288 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:02.854339 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:02.854444 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 06:19:02.854482 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":107,"threads_started":1}
I20260926 06:19:02.854451 21039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:02.850391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.194:34907 (local address 127.0.0.1:50026)
0926 06:19:02.850603 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:02.850612 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:02.850650 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:02.851070 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:19:02.851071 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:19:02.851073 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:19:02.851119 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:02.851121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:02.852335 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:02.852336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:02.853157 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:02.853159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:02.854120 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:02.854121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:02.854200 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:02.854215 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 06:19:02.854293 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 06:19:02.854347 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":4,"thread_start_us":68,"threads_started":1}
I20260926 06:19:02.854660 21040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:02.850616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50026 (local address 127.2.110.194:34907)
0926 06:19:02.850813 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:02.850817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:02.850825 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:19:02.850878 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:02.850881 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:02.850896 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:02.850990 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:19:02.851200 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:02.852255 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:02.853286 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:02.853829 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:02.853926 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:02.854073 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:02.854529 (+ 456us) server_negotiation.cc:299] Negotiation successful
0926 06:19:02.854582 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":86,"threads_started":1}
I20260926 06:19:02.855647 20690 tablet_service.cc:1467] Processing CreateTablet for tablet 088e055debd140909488ef3aec8b0103 (DEFAULT_TABLE table=test-table [id=4cf59349c38745cca5300f5aba83a611]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:19:02.855679 20824 tablet_service.cc:1467] Processing CreateTablet for tablet 088e055debd140909488ef3aec8b0103 (DEFAULT_TABLE table=test-table [id=4cf59349c38745cca5300f5aba83a611]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:19:02.855971 20690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 088e055debd140909488ef3aec8b0103. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:19:02.856011 20824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 088e055debd140909488ef3aec8b0103. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:19:02.857725 21035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:02.850136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.195:35011 (local address 127.0.0.1:59458)
0926 06:19:02.850280 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:02.850350 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:02.850420 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:02.850707 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:19:02.850710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:19:02.850713 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:19:02.850807 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:02.850809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:02.852237 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:02.852237 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:02.853815 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:02.853817 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:02.857439 (+ 3622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:02.857440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:02.857512 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:02.857516 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:19:02.857583 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 06:19:02.857632 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":39,"thread_start_us":89,"threads_started":1}
I20260926 06:19:02.858089 21038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:02.850337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59458 (local address 127.2.110.195:35011)
0926 06:19:02.850537 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:02.850542 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:02.850552 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:19:02.850594 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:02.850597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:02.850612 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:02.850690 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:19:02.850890 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:02.852148 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:02.853916 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:02.855117 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:02.855193 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:02.855250 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:02.857631 (+ 2381us) server_negotiation.cc:299] Negotiation successful
0926 06:19:02.857692 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":86,"threads_started":1}
I20260926 06:19:02.858961 21041 tablet_bootstrap.cc:492] T 088e055debd140909488ef3aec8b0103 P 6e9cff8d0011404a9cdfddb007b6db95: Bootstrap starting.
I20260926 06:19:02.858870 20958 tablet_service.cc:1467] Processing CreateTablet for tablet 088e055debd140909488ef3aec8b0103 (DEFAULT_TABLE table=test-table [id=4cf59349c38745cca5300f5aba83a611]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:19:02.859221 20958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 088e055debd140909488ef3aec8b0103. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:19:02.859555 21041 tablet_bootstrap.cc:654] T 088e055debd140909488ef3aec8b0103 P 6e9cff8d0011404a9cdfddb007b6db95: Neither blocks nor log segments found. Creating new log.
I20260926 06:19:02.859989 21041 log.cc:824] T 088e055debd140909488ef3aec8b0103 P 6e9cff8d0011404a9cdfddb007b6db95: Log is configured to *not* fsync() on all Append() calls
I20260926 06:19:02.860322 21042 tablet_bootstrap.cc:492] T 088e055debd140909488ef3aec8b0103 P ddac8cd8c066434f81c299fdcc212f7d: Bootstrap starting.
I20260926 06:19:02.860778 21041 tablet_bootstrap.cc:492] T 088e055debd140909488ef3aec8b0103 P 6e9cff8d0011404a9cdfddb007b6db95: No bootstrap required, opened a new log
I20260926 06:19:02.860848 21041 ts_tablet_manager.cc:1397] T 088e055debd140909488ef3aec8b0103 P 6e9cff8d0011404a9cdfddb007b6db95: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 06:19:02.860862 21042 tablet_bootstrap.cc:654] T 088e055debd140909488ef3aec8b0103 P ddac8cd8c066434f81c299fdcc212f7d: Neither blocks nor log segments found. Creating new log.
I20260926 06:19:02.861250 21042 log.cc:824] T 088e055debd140909488ef3aec8b0103 P ddac8cd8c066434f81c299fdcc212f7d: Log is configured to *not* fsync() on all Append() calls
I20260926 06:19:02.862218 21042 tablet_bootstrap.cc:492] T 088e055debd140909488ef3aec8b0103 P ddac8cd8c066434f81c299fdcc212f7d: No bootstrap required, opened a new log
I20260926 06:19:02.862208 21045 tablet_bootstrap.cc:492] T 088e055debd140909488ef3aec8b0103 P 264d75b1ad55441eab6a9f1a43f4b218: Bootstrap starting.
I20260926 06:19:02.862286 21042 ts_tablet_manager.cc:1397] T 088e055debd140909488ef3aec8b0103 P ddac8cd8c066434f81c299fdcc212f7d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 06:19:02.862859 21045 tablet_bootstrap.cc:654] T 088e055debd140909488ef3aec8b0103 P 264d75b1ad55441eab6a9f1a43f4b218: Neither blocks nor log segments found. Creating new log.
I20260926 06:19:02.863080 21041 raft_consensus.cc:348] T 088e055debd140909488ef3aec8b0103 P 6e9cff8d0011404a9cdfddb007b6db95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "264d75b1ad55441eab6a9f1a43f4b218" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 35011 } } peers { permanent_uuid: "ddac8cd8c066434f81c299fdcc212f7d" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 34907 } } peers { permanent_uuid: "6e9cff8d0011404a9cdfddb007b6db95" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 36689 } }
I20260926 06:19:02.863308 21041 raft_consensus.cc:374] T 088e055debd140909488ef3aec8b0103 P 6e9cff8d0011404a9cdfddb007b6db95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:19:02.863349 21041 raft_consensus.cc:729] T 088e055debd140909488ef3aec8b0103 P 6e9cff8d0011404a9cdfddb007b6db95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e9cff8d0011404a9cdfddb007b6db95, State: Initialized, Role: FOLLOWER
I20260926 06:19:02.863523 21045 log.cc:824] T 088e055debd140909488ef3aec8b0103 P 264d75b1ad55441eab6a9f1a43f4b218: Log is configured to *not* fsync() on all Append() calls
I20260926 06:19:02.863489 21041 consensus_queue.cc:260] T 088e055debd140909488ef3aec8b0103 P 6e9cff8d0011404a9cdfddb007b6db95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "264d75b1ad55441eab6a9f1a43f4b218" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 35011 } } peers { permanent_uuid: "ddac8cd8c066434f81c299fdcc212f7d" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 34907 } } peers { permanent_uuid: "6e9cff8d0011404a9cdfddb007b6db95" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 36689 } }
I20260926 06:19:02.863783 21041 ts_tablet_manager.cc:1428] T 088e055debd140909488ef3aec8b0103 P 6e9cff8d0011404a9cdfddb007b6db95: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 06:19:02.863935 20756 heartbeater.cc:503] Master 127.2.110.254:33997 was elected leader, sending a full tablet report...
I20260926 06:19:02.864141 21042 raft_consensus.cc:348] T 088e055debd140909488ef3aec8b0103 P ddac8cd8c066434f81c299fdcc212f7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "264d75b1ad55441eab6a9f1a43f4b218" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 35011 } } peers { permanent_uuid: "ddac8cd8c066434f81c299fdcc212f7d" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 34907 } } peers { permanent_uuid: "6e9cff8d0011404a9cdfddb007b6db95" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 36689 } }
I20260926 06:19:02.864336 21042 raft_consensus.cc:374] T 088e055debd140909488ef3aec8b0103 P ddac8cd8c066434f81c299fdcc212f7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:19:02.864394 21042 raft_consensus.cc:729] T 088e055debd140909488ef3aec8b0103 P ddac8cd8c066434f81c299fdcc212f7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddac8cd8c066434f81c299fdcc212f7d, State: Initialized, Role: FOLLOWER
I20260926 06:19:02.864502 21042 consensus_queue.cc:260] T 088e055debd140909488ef3aec8b0103 P ddac8cd8c066434f81c299fdcc212f7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "264d75b1ad55441eab6a9f1a43f4b218" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 35011 } } peers { permanent_uuid: "ddac8cd8c066434f81c299fdcc212f7d" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 34907 } } peers { permanent_uuid: "6e9cff8d0011404a9cdfddb007b6db95" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 36689 } }
I20260926 06:19:02.864595 21045 tablet_bootstrap.cc:492] T 088e055debd140909488ef3aec8b0103 P 264d75b1ad55441eab6a9f1a43f4b218: No bootstrap required, opened a new log
I20260926 06:19:02.864665 21045 ts_tablet_manager.cc:1397] T 088e055debd140909488ef3aec8b0103 P 264d75b1ad55441eab6a9f1a43f4b218: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 06:19:02.864761 21042 ts_tablet_manager.cc:1428] T 088e055debd140909488ef3aec8b0103 P ddac8cd8c066434f81c299fdcc212f7d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 06:19:02.864913 20890 heartbeater.cc:503] Master 127.2.110.254:33997 was elected leader, sending a full tablet report...
I20260926 06:19:02.866489 21045 raft_consensus.cc:348] T 088e055debd140909488ef3aec8b0103 P 264d75b1ad55441eab6a9f1a43f4b218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "264d75b1ad55441eab6a9f1a43f4b218" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 35011 } } peers { permanent_uuid: "ddac8cd8c066434f81c299fdcc212f7d" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 34907 } } peers { permanent_uuid: "6e9cff8d0011404a9cdfddb007b6db95" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 36689 } }
I20260926 06:19:02.866649 21045 raft_consensus.cc:374] T 088e055debd140909488ef3aec8b0103 P 264d75b1ad55441eab6a9f1a43f4b218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:19:02.866698 21045 raft_consensus.cc:729] T 088e055debd140909488ef3aec8b0103 P 264d75b1ad55441eab6a9f1a43f4b218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 264d75b1ad55441eab6a9f1a43f4b218, State: Initialized, Role: FOLLOWER
I20260926 06:19:02.866822 21045 consensus_queue.cc:260] T 088e055debd140909488ef3aec8b0103 P 264d75b1ad55441eab6a9f1a43f4b218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "264d75b1ad55441eab6a9f1a43f4b218" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 35011 } } peers { permanent_uuid: "ddac8cd8c066434f81c299fdcc212f7d" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 34907 } } peers { permanent_uuid: "6e9cff8d0011404a9cdfddb007b6db95" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 36689 } }
I20260926 06:19:02.867081 21045 ts_tablet_manager.cc:1428] T 088e055debd140909488ef3aec8b0103 P 264d75b1ad55441eab6a9f1a43f4b218: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 06:19:02.867161 21024 heartbeater.cc:503] Master 127.2.110.254:33997 was elected leader, sending a full tablet report...
W20260926 06:19:02.898556 20891 tablet.cc:2369] T 088e055debd140909488ef3aec8b0103 P ddac8cd8c066434f81c299fdcc212f7d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:19:02.921705 21046 raft_consensus.cc:482] T 088e055debd140909488ef3aec8b0103 P 6e9cff8d0011404a9cdfddb007b6db95 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:19:02.921852 21046 raft_consensus.cc:504] T 088e055debd140909488ef3aec8b0103 P 6e9cff8d0011404a9cdfddb007b6db95 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "264d75b1ad55441eab6a9f1a43f4b218" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 35011 } } peers { permanent_uuid: "ddac8cd8c066434f81c299fdcc212f7d" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 34907 } } peers { permanent_uuid: "6e9cff8d0011404a9cdfddb007b6db95" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 36689 } }
I20260926 06:19:02.922257 21046 leader_election.cc:288] T 088e055debd140909488ef3aec8b0103 P 6e9cff8d0011404a9cdfddb007b6db95 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 264d75b1ad55441eab6a9f1a43f4b218 (127.2.110.195:35011), ddac8cd8c066434f81c299fdcc212f7d (127.2.110.194:34907)
I20260926 06:19:02.926546 21038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:02.922610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:54073 (local address 127.2.110.195:35011)
0926 06:19:02.922664 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:02.922667 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:02.922678 (+ 11us) server_negotiation.cc:407] Connection header received
0926 06:19:02.922712 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:02.922713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:02.922723 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:02.922881 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:19:02.923083 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:02.923889 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:02.925487 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:02.926083 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:02.926172 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:02.926182 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:02.926422 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 06:19:02.926474 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 06:19:02.926546 21040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:02.922724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:52077 (local address 127.2.110.194:34907)
0926 06:19:02.922767 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:02.922769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:02.922876 (+ 107us) server_negotiation.cc:407] Connection header received
0926 06:19:02.922927 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:02.922928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:02.922938 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:02.923023 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:19:02.923213 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:02.923915 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:02.925487 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:02.926083 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:02.926173 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:02.926182 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:02.926422 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 06:19:02.926474 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 06:19:02.926549 20758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:02.922490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.195:35011 (local address 127.2.110.193:54073)
0926 06:19:02.922551 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:02.922564 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:02.922649 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:02.922891 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:19:02.922892 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:19:02.922894 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:19:02.922969 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:02.922971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:02.924010 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:02.924011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:02.925366 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:02.925371 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:02.926207 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:02.926209 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:02.926326 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:02.926328 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 06:19:02.926383 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 06:19:02.926453 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":55}
I20260926 06:19:02.926556 21050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:02.922624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.194:34907 (local address 127.2.110.193:52077)
0926 06:19:02.922807 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:02.922817 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:02.922870 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:02.923032 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:19:02.923033 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:19:02.923035 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:19:02.923111 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:02.923117 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:02.924010 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:02.924011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:02.925378 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:02.925380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:02.926210 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:02.926211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:02.926305 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:02.926310 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 06:19:02.926388 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 06:19:02.926452 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":88,"thread_start_us":122,"threads_started":1}
I20260926 06:19:02.926967 20978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "088e055debd140909488ef3aec8b0103" candidate_uuid: "6e9cff8d0011404a9cdfddb007b6db95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "264d75b1ad55441eab6a9f1a43f4b218" is_pre_election: true
I20260926 06:19:02.926967 20832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "088e055debd140909488ef3aec8b0103" candidate_uuid: "6e9cff8d0011404a9cdfddb007b6db95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddac8cd8c066434f81c299fdcc212f7d" is_pre_election: true
I20260926 06:19:02.927132 20978 raft_consensus.cc:2454] T 088e055debd140909488ef3aec8b0103 P 264d75b1ad55441eab6a9f1a43f4b218 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e9cff8d0011404a9cdfddb007b6db95 in term 0.
I20260926 06:19:02.927132 20832 raft_consensus.cc:2454] T 088e055debd140909488ef3aec8b0103 P ddac8cd8c066434f81c299fdcc212f7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e9cff8d0011404a9cdfddb007b6db95 in term 0.
I20260926 06:19:02.927374 20645 leader_election.cc:302] T 088e055debd140909488ef3aec8b0103 P 6e9cff8d0011404a9cdfddb007b6db95 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e9cff8d0011404a9cdfddb007b6db95, ddac8cd8c066434f81c299fdcc212f7d; no voters:
I20260926 06:19:02.927534 21046 raft_consensus.cc:2781] T 088e055debd140909488ef3aec8b0103 P 6e9cff8d0011404a9cdfddb007b6db95 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:19:02.927590 21046 raft_consensus.cc:482] T 088e055debd140909488ef3aec8b0103 P 6e9cff8d0011404a9cdfddb007b6db95 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:19:02.927616 21046 raft_consensus.cc:3037] T 088e055debd140909488ef3aec8b0103 P 6e9cff8d0011404a9cdfddb007b6db95 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:19:02.928354 21046 raft_consensus.cc:504] T 088e055debd140909488ef3aec8b0103 P 6e9cff8d0011404a9cdfddb007b6db95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "264d75b1ad55441eab6a9f1a43f4b218" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 35011 } } peers { permanent_uuid: "ddac8cd8c066434f81c299fdcc212f7d" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 34907 } } peers { permanent_uuid: "6e9cff8d0011404a9cdfddb007b6db95" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 36689 } }
I20260926 06:19:02.928498 21046 leader_election.cc:288] T 088e055debd140909488ef3aec8b0103 P 6e9cff8d0011404a9cdfddb007b6db95 [CANDIDATE]: Term 1 election: Requested vote from peers 264d75b1ad55441eab6a9f1a43f4b218 (127.2.110.195:35011), ddac8cd8c066434f81c299fdcc212f7d (127.2.110.194:34907)
I20260926 06:19:02.928704 20832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "088e055debd140909488ef3aec8b0103" candidate_uuid: "6e9cff8d0011404a9cdfddb007b6db95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddac8cd8c066434f81c299fdcc212f7d"
I20260926 06:19:02.928705 20978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "088e055debd140909488ef3aec8b0103" candidate_uuid: "6e9cff8d0011404a9cdfddb007b6db95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "264d75b1ad55441eab6a9f1a43f4b218"
I20260926 06:19:02.928787 20832 raft_consensus.cc:3037] T 088e055debd140909488ef3aec8b0103 P ddac8cd8c066434f81c299fdcc212f7d [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:19:02.928807 20978 raft_consensus.cc:3037] T 088e055debd140909488ef3aec8b0103 P 264d75b1ad55441eab6a9f1a43f4b218 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:19:02.929524 20832 raft_consensus.cc:2454] T 088e055debd140909488ef3aec8b0103 P ddac8cd8c066434f81c299fdcc212f7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e9cff8d0011404a9cdfddb007b6db95 in term 1.
I20260926 06:19:02.929692 20645 leader_election.cc:302] T 088e055debd140909488ef3aec8b0103 P 6e9cff8d0011404a9cdfddb007b6db95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e9cff8d0011404a9cdfddb007b6db95, ddac8cd8c066434f81c299fdcc212f7d; no voters:
I20260926 06:19:02.929746 20978 raft_consensus.cc:2454] T 088e055debd140909488ef3aec8b0103 P 264d75b1ad55441eab6a9f1a43f4b218 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e9cff8d0011404a9cdfddb007b6db95 in term 1.
I20260926 06:19:02.929818 21046 raft_consensus.cc:2781] T 088e055debd140909488ef3aec8b0103 P 6e9cff8d0011404a9cdfddb007b6db95 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:19:02.930012 21046 raft_consensus.cc:686] T 088e055debd140909488ef3aec8b0103 P 6e9cff8d0011404a9cdfddb007b6db95 [term 1 LEADER]: Becoming Leader. State: Replica: 6e9cff8d0011404a9cdfddb007b6db95, State: Running, Role: LEADER
I20260926 06:19:02.930126 21046 consensus_queue.cc:237] T 088e055debd140909488ef3aec8b0103 P 6e9cff8d0011404a9cdfddb007b6db95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "264d75b1ad55441eab6a9f1a43f4b218" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 35011 } } peers { permanent_uuid: "ddac8cd8c066434f81c299fdcc212f7d" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 34907 } } peers { permanent_uuid: "6e9cff8d0011404a9cdfddb007b6db95" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 36689 } }
I20260926 06:19:02.931056 20565 catalog_manager.cc:5495] T 088e055debd140909488ef3aec8b0103 P 6e9cff8d0011404a9cdfddb007b6db95 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e9cff8d0011404a9cdfddb007b6db95 (127.2.110.193). New cstate: current_term: 1 leader_uuid: "6e9cff8d0011404a9cdfddb007b6db95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "264d75b1ad55441eab6a9f1a43f4b218" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 35011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddac8cd8c066434f81c299fdcc212f7d" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 34907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e9cff8d0011404a9cdfddb007b6db95" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 36689 } health_report { overall_health: HEALTHY } } }
I20260926 06:19:02.951215 21037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:02.948455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35038 (local address 127.2.110.193:36689)
0926 06:19:02.948540 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:02.948542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:02.948551 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:19:02.948594 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:02.948599 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:02.948618 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:02.948698 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:19:02.948844 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:02.949461 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:02.950309 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:02.950581 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:02.950681 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:02.950835 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:19:02.950948 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:19:02.951062 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:02.951093 (+ 31us) server_negotiation.cc:299] Negotiation successful
0926 06:19:02.951105 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 06:19:02.953711 20832 raft_consensus.cc:1261] T 088e055debd140909488ef3aec8b0103 P ddac8cd8c066434f81c299fdcc212f7d [term 1 FOLLOWER]: Refusing update from remote peer 6e9cff8d0011404a9cdfddb007b6db95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:19:02.953711 20978 raft_consensus.cc:1261] T 088e055debd140909488ef3aec8b0103 P 264d75b1ad55441eab6a9f1a43f4b218 [term 1 FOLLOWER]: Refusing update from remote peer 6e9cff8d0011404a9cdfddb007b6db95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:19:02.954217 21051 consensus_queue.cc:1035] T 088e055debd140909488ef3aec8b0103 P 6e9cff8d0011404a9cdfddb007b6db95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddac8cd8c066434f81c299fdcc212f7d" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 34907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:19:02.954442 21046 consensus_queue.cc:1035] T 088e055debd140909488ef3aec8b0103 P 6e9cff8d0011404a9cdfddb007b6db95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "264d75b1ad55441eab6a9f1a43f4b218" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 35011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:19:02.957494 21056 mvcc.cc:205] Tried to move back new op lower bound from 7333492911933505536 to 7333492911842553856. Current Snapshot: MvccSnapshot[applied={T|T < 7333492911933505536}]
I20260926 06:19:02.958388 21055 mvcc.cc:205] Tried to move back new op lower bound from 7333492911933505536 to 7333492911842553856. Current Snapshot: MvccSnapshot[applied={T|T < 7333492911933505536 or (T in {7333492911933505536})}]
I20260926 06:19:02.960857 20564 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:19:02.960980 20564 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:19:02.961789 20564 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cf816272cb064d8499deb48ac222d5f7: Sending DeleteTablet for 3 replicas of tablet 088e055debd140909488ef3aec8b0103
I20260926 06:19:02.962193 20824 tablet_service.cc:1514] Processing DeleteTablet for tablet 088e055debd140909488ef3aec8b0103 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:19:02 UTC) from {username='kudu', principal='kudu/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:50026
I20260926 06:19:02.962222 20958 tablet_service.cc:1514] Processing DeleteTablet for tablet 088e055debd140909488ef3aec8b0103 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:19:02 UTC) from {username='kudu', principal='kudu/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:59458
I20260926 06:19:02.962445 21064 tablet_replica.cc:330] T 088e055debd140909488ef3aec8b0103 P ddac8cd8c066434f81c299fdcc212f7d: stopping tablet replica
I20260926 06:19:02.962460 20690 tablet_service.cc:1514] Processing DeleteTablet for tablet 088e055debd140909488ef3aec8b0103 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:19:02 UTC) from {username='kudu', principal='kudu/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:35032
I20260926 06:19:02.962558 21064 raft_consensus.cc:2229] T 088e055debd140909488ef3aec8b0103 P ddac8cd8c066434f81c299fdcc212f7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:19:02.962679 21064 raft_consensus.cc:2258] T 088e055debd140909488ef3aec8b0103 P ddac8cd8c066434f81c299fdcc212f7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:19:02.962908 21066 tablet_replica.cc:330] T 088e055debd140909488ef3aec8b0103 P 6e9cff8d0011404a9cdfddb007b6db95: stopping tablet replica
I20260926 06:19:02.962929 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 20627
I20260926 06:19:02.963824 21064 ts_tablet_manager.cc:1905] T 088e055debd140909488ef3aec8b0103 P ddac8cd8c066434f81c299fdcc212f7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:19:02.963953 21065 tablet_replica.cc:330] T 088e055debd140909488ef3aec8b0103 P 264d75b1ad55441eab6a9f1a43f4b218: stopping tablet replica
I20260926 06:19:02.964046 21065 raft_consensus.cc:2229] T 088e055debd140909488ef3aec8b0103 P 264d75b1ad55441eab6a9f1a43f4b218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:19:02.964128 21065 raft_consensus.cc:2258] T 088e055debd140909488ef3aec8b0103 P 264d75b1ad55441eab6a9f1a43f4b218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:19:02.964529 21065 ts_tablet_manager.cc:1905] T 088e055debd140909488ef3aec8b0103 P 264d75b1ad55441eab6a9f1a43f4b218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:19:02.965988 21064 ts_tablet_manager.cc:1918] T 088e055debd140909488ef3aec8b0103 P ddac8cd8c066434f81c299fdcc212f7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:19:02.966058 21064 log.cc:1196] T 088e055debd140909488ef3aec8b0103 P ddac8cd8c066434f81c299fdcc212f7d: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/ts-1/wal/wals/088e055debd140909488ef3aec8b0103
I20260926 06:19:02.966121 21065 ts_tablet_manager.cc:1918] T 088e055debd140909488ef3aec8b0103 P 264d75b1ad55441eab6a9f1a43f4b218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:19:02.966188 21065 log.cc:1196] T 088e055debd140909488ef3aec8b0103 P 264d75b1ad55441eab6a9f1a43f4b218: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403292369990-2491-0/minicluster-data/ts-2/wal/wals/088e055debd140909488ef3aec8b0103
I20260926 06:19:02.966347 21064 ts_tablet_manager.cc:1939] T 088e055debd140909488ef3aec8b0103 P ddac8cd8c066434f81c299fdcc212f7d: Deleting consensus metadata
I20260926 06:19:02.966476 21065 ts_tablet_manager.cc:1939] T 088e055debd140909488ef3aec8b0103 P 264d75b1ad55441eab6a9f1a43f4b218: Deleting consensus metadata
I20260926 06:19:02.966811 20551 catalog_manager.cc:4841] TS ddac8cd8c066434f81c299fdcc212f7d (127.2.110.194:34907): tablet 088e055debd140909488ef3aec8b0103 (table test-table [id=4cf59349c38745cca5300f5aba83a611]) successfully deleted
I20260926 06:19:02.966955 20551 catalog_manager.cc:4841] TS 264d75b1ad55441eab6a9f1a43f4b218 (127.2.110.195:35011): tablet 088e055debd140909488ef3aec8b0103 (table test-table [id=4cf59349c38745cca5300f5aba83a611]) successfully deleted
W20260926 06:19:02.970008 20550 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.110.193:36689 (error 108)
I20260926 06:19:02.970181 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 20761
I20260926 06:19:02.972103 21035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:02.971883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.193:36689 (local address 127.0.0.1:35044)
0926 06:19:02.971970 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:02.972069 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.193:36689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 06:19:02.972370 20550 catalog_manager.cc:4584] TS 6e9cff8d0011404a9cdfddb007b6db95 (127.2.110.193:36689): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 088e055debd140909488ef3aec8b0103: Network error: Client connection negotiation failed: client connection to 127.2.110.193:36689: connect: Connection refused (error 111)
I20260926 06:19:02.975593 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 20895
I20260926 06:19:02.980464 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 20533
2026-09-26T06:19:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2052 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T06:19:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:19:03Z Disabled control of system clock
I20260926 06:19:03.009704 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:36871
--webserver_interface=127.2.110.254
--webserver_port=0
--builtin_ntp_servers=127.2.110.212:38095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.110.254:36871
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 06:19:03.093439 21071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:19:03.093678 21071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:19:03.093701 21071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:19:03.094834 21071 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:19:03.094874 21071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:19:03.094887 21071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:19:03.094901 21071 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:19:03.094913 21071 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:19:03.096235 21071 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:38095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.110.254:36871
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:36871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:19:03.096469 21071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:19:03.096721 21071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:19:03.101783 21077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:03.101914 21071 server_base.cc:1034] running on GCE node
W20260926 06:19:03.101783 21079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:03.101879 21076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:03.102301 21071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:19:03.102636 21071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:19:03.103790 21071 hybrid_clock.cc:648] HybridClock initialized: now 1790403543103767 us; error 42 us; skew 500 ppm
I20260926 06:19:03.108515 21071 webserver.cc:460] Webserver started at http://127.2.110.254:37555/ using document root <none> and password file <none>
I20260926 06:19:03.108835 21071 fs_manager.cc:359] Metadata directory not provided
I20260926 06:19:03.108915 21071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:19:03.109077 21071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:19:03.110321 21071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/master-0/data/instance:
uuid: "d591889470a342648ea5f73d810a7754"
format_stamp: "Formatted at 2026-09-26 06:19:03 on dist-test-slave-hndd"
server_key: "091d7b34a321de04c25968a01508fe85"
server_key_iv: "c9542c1778164c77476c3e4a2b59d027"
server_key_version: "encryptionkey@0"
I20260926 06:19:03.110692 21071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/master-0/wal/instance:
uuid: "d591889470a342648ea5f73d810a7754"
format_stamp: "Formatted at 2026-09-26 06:19:03 on dist-test-slave-hndd"
server_key: "091d7b34a321de04c25968a01508fe85"
server_key_iv: "c9542c1778164c77476c3e4a2b59d027"
server_key_version: "encryptionkey@0"
I20260926 06:19:03.112092 21071 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 06:19:03.112950 21085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:03.113183 21071 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:19:03.113296 21071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "d591889470a342648ea5f73d810a7754"
format_stamp: "Formatted at 2026-09-26 06:19:03 on dist-test-slave-hndd"
server_key: "091d7b34a321de04c25968a01508fe85"
server_key_iv: "c9542c1778164c77476c3e4a2b59d027"
server_key_version: "encryptionkey@0"
I20260926 06:19:03.113458 21071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:19:03.122960 21071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:19:03.123394 21071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:19:03.123574 21071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:19:03.128288 21071 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.254:36871
I20260926 06:19:03.128322 21137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.254:36871 every 8 connection(s)
I20260926 06:19:03.128723 21071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:19:03.129982 21138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:19:03.132356 21138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d591889470a342648ea5f73d810a7754: Bootstrap starting.
I20260926 06:19:03.133008 21138 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d591889470a342648ea5f73d810a7754: Neither blocks nor log segments found. Creating new log.
I20260926 06:19:03.133322 21138 log.cc:824] T 00000000000000000000000000000000 P d591889470a342648ea5f73d810a7754: Log is configured to *not* fsync() on all Append() calls
I20260926 06:19:03.134155 21138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d591889470a342648ea5f73d810a7754: No bootstrap required, opened a new log
I20260926 06:19:03.134684 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 21071
I20260926 06:19:03.134825 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/master-0/wal/instance
I20260926 06:19:03.135037 2491 external_mini_cluster.cc:1428] Setting key 2337511e890bf42ee873428a3f22d4af
I20260926 06:19:03.136538 21138 raft_consensus.cc:348] T 00000000000000000000000000000000 P d591889470a342648ea5f73d810a7754 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d591889470a342648ea5f73d810a7754" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 36871 } }
I20260926 06:19:03.136722 21138 raft_consensus.cc:374] T 00000000000000000000000000000000 P d591889470a342648ea5f73d810a7754 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:19:03.136765 21138 raft_consensus.cc:729] T 00000000000000000000000000000000 P d591889470a342648ea5f73d810a7754 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d591889470a342648ea5f73d810a7754, State: Initialized, Role: FOLLOWER
I20260926 06:19:03.136888 21138 consensus_queue.cc:260] T 00000000000000000000000000000000 P d591889470a342648ea5f73d810a7754 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d591889470a342648ea5f73d810a7754" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 36871 } }
I20260926 06:19:03.136999 21138 raft_consensus.cc:388] T 00000000000000000000000000000000 P d591889470a342648ea5f73d810a7754 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:19:03.137033 21138 raft_consensus.cc:482] T 00000000000000000000000000000000 P d591889470a342648ea5f73d810a7754 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:19:03.137085 21138 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d591889470a342648ea5f73d810a7754 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:19:03.138034 21138 raft_consensus.cc:504] T 00000000000000000000000000000000 P d591889470a342648ea5f73d810a7754 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d591889470a342648ea5f73d810a7754" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 36871 } }
I20260926 06:19:03.138156 21138 leader_election.cc:302] T 00000000000000000000000000000000 P d591889470a342648ea5f73d810a7754 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d591889470a342648ea5f73d810a7754; no voters:
I20260926 06:19:03.138324 21138 leader_election.cc:288] T 00000000000000000000000000000000 P d591889470a342648ea5f73d810a7754 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:19:03.138399 21143 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d591889470a342648ea5f73d810a7754 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:19:03.138581 21138 sys_catalog.cc:564] T 00000000000000000000000000000000 P d591889470a342648ea5f73d810a7754 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:19:03.138593 21143 raft_consensus.cc:686] T 00000000000000000000000000000000 P d591889470a342648ea5f73d810a7754 [term 1 LEADER]: Becoming Leader. State: Replica: d591889470a342648ea5f73d810a7754, State: Running, Role: LEADER
I20260926 06:19:03.138875 21143 consensus_queue.cc:237] T 00000000000000000000000000000000 P d591889470a342648ea5f73d810a7754 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d591889470a342648ea5f73d810a7754" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 36871 } }
I20260926 06:19:03.139312 21145 sys_catalog.cc:455] T 00000000000000000000000000000000 P d591889470a342648ea5f73d810a7754 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d591889470a342648ea5f73d810a7754. Latest consensus state: current_term: 1 leader_uuid: "d591889470a342648ea5f73d810a7754" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d591889470a342648ea5f73d810a7754" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 36871 } } }
I20260926 06:19:03.139393 21145 sys_catalog.cc:458] T 00000000000000000000000000000000 P d591889470a342648ea5f73d810a7754 [sys.catalog]: This master's current role is: LEADER
I20260926 06:19:03.139706 21144 sys_catalog.cc:455] T 00000000000000000000000000000000 P d591889470a342648ea5f73d810a7754 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d591889470a342648ea5f73d810a7754" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d591889470a342648ea5f73d810a7754" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 36871 } } }
I20260926 06:19:03.139786 21144 sys_catalog.cc:458] T 00000000000000000000000000000000 P d591889470a342648ea5f73d810a7754 [sys.catalog]: This master's current role is: LEADER
I20260926 06:19:03.140013 21152 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:19:03.140594 21152 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:19:03.142377 21142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:03.135634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38732 (local address 127.2.110.254:36871)
0926 06:19:03.135874 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:03.135883 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:03.135921 (+ 38us) server_negotiation.cc:407] Connection header received
0926 06:19:03.136483 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:03.136492 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:03.136575 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:03.136732 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:03.137023 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:03.138909 (+ 1886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:03.140136 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:03.140710 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:03.140831 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:03.141603 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:03.141607 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:03.141607 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:19:03.141820 (+ 213us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:03.141852 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:19:03.141863 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:03.141952 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:03.142067 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 06:19:03.142152 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":100,"threads_started":1}
I20260926 06:19:03.142892 21152 catalog_manager.cc:1347] Generated new cluster ID: 1c57b7a4aa944a36b49de52f3e2af36c
I20260926 06:19:03.143011 21152 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:19:03.164191 21152 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:19:03.164934 21152 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:19:03.168380 21152 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d591889470a342648ea5f73d810a7754: Generated new TSK 0
I20260926 06:19:03.168645 21152 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:19:03.171037 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.193:0
--local_ip_for_outbound_sockets=127.2.110.193
--webserver_interface=127.2.110.193
--webserver_port=0
--tserver_master_addrs=127.2.110.254:36871
--builtin_ntp_servers=127.2.110.212:38095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 06:19:03.258423 21162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:19:03.258759 21162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:19:03.258800 21162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:19:03.261708 21162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:19:03.261813 21162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:19:03.261857 21162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.193
I20260926 06:19:03.264657 21162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:38095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.110.193
--webserver_port=0
--tserver_master_addrs=127.2.110.254:36871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.193
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:19:03.265027 21162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:19:03.265429 21162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:19:03.271672 21170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:03.271652 21168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:03.271898 21162 server_base.cc:1034] running on GCE node
W20260926 06:19:03.271857 21167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:03.272159 21162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:19:03.272429 21162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:19:03.273617 21162 hybrid_clock.cc:648] HybridClock initialized: now 1790403543273594 us; error 37 us; skew 500 ppm
I20260926 06:19:03.275086 21162 webserver.cc:460] Webserver started at http://127.2.110.193:40017/ using document root <none> and password file <none>
I20260926 06:19:03.275345 21162 fs_manager.cc:359] Metadata directory not provided
I20260926 06:19:03.275424 21162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:19:03.275576 21162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:19:03.276813 21162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/ts-0/data/instance:
uuid: "975696e97c9b474ea0b457fa719ef421"
format_stamp: "Formatted at 2026-09-26 06:19:03 on dist-test-slave-hndd"
server_key: "49a35a2579ebb53390f91aec2811f137"
server_key_iv: "b0c83eaf447860a3d2d605f18a2e267c"
server_key_version: "encryptionkey@0"
I20260926 06:19:03.277321 21162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/ts-0/wal/instance:
uuid: "975696e97c9b474ea0b457fa719ef421"
format_stamp: "Formatted at 2026-09-26 06:19:03 on dist-test-slave-hndd"
server_key: "49a35a2579ebb53390f91aec2811f137"
server_key_iv: "b0c83eaf447860a3d2d605f18a2e267c"
server_key_version: "encryptionkey@0"
I20260926 06:19:03.279110 21162 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:19:03.280403 21176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:03.280695 21162 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260926 06:19:03.280834 21162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/ts-0/wal
uuid: "975696e97c9b474ea0b457fa719ef421"
format_stamp: "Formatted at 2026-09-26 06:19:03 on dist-test-slave-hndd"
server_key: "49a35a2579ebb53390f91aec2811f137"
server_key_iv: "b0c83eaf447860a3d2d605f18a2e267c"
server_key_version: "encryptionkey@0"
I20260926 06:19:03.280939 21162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:19:03.293488 21162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:19:03.293835 21162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:19:03.293998 21162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:19:03.294256 21162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:19:03.294615 21162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:19:03.294670 21162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:03.294715 21162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:19:03.294749 21162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:03.302443 21162 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.193:33599
I20260926 06:19:03.302488 21289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.193:33599 every 8 connection(s)
I20260926 06:19:03.302845 21162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
I20260926 06:19:03.306574 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 21162
I20260926 06:19:03.306672 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/ts-0/wal/instance
I20260926 06:19:03.307039 2491 external_mini_cluster.cc:1428] Setting key 6389700f53c19f19bad330c6023bdb1d
I20260926 06:19:03.308187 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.194:0
--local_ip_for_outbound_sockets=127.2.110.194
--webserver_interface=127.2.110.194
--webserver_port=0
--tserver_master_addrs=127.2.110.254:36871
--builtin_ntp_servers=127.2.110.212:38095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 06:19:03.309003 21142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:03.303728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:47097 (local address 127.2.110.254:36871)
0926 06:19:03.303805 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:03.303809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:03.304057 (+ 248us) server_negotiation.cc:407] Connection header received
0926 06:19:03.304526 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:03.304527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:03.304544 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:03.304619 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:03.305292 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:03.305845 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:03.307071 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:03.307442 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:03.307540 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:03.308670 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:03.308672 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:03.308672 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:19:03.308678 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:03.308694 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:19:03.308699 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:03.308786 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:03.308882 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 06:19:03.308915 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 06:19:03.309273 21292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:03.303644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:36871 (local address 127.2.110.193:47097)
0926 06:19:03.303906 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:03.303946 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:03.304434 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:03.304671 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:19:03.304674 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:19:03.304682 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:19:03.305184 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:03.305188 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:03.305961 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:03.305962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:03.306950 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:03.306953 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:03.307552 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:03.307553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:03.307709 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:03.308480 (+ 771us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:19:03.308481 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:19:03.308525 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:19:03.308553 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:19:03.308792 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:19:03.308798 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:19:03.308801 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:19:03.308883 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 06:19:03.309005 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":95,"threads_started":1}
I20260926 06:19:03.309819 21290 heartbeater.cc:347] Connected to a master server at 127.2.110.254:36871
I20260926 06:19:03.309923 21290 heartbeater.cc:464] Registering TS with master...
I20260926 06:19:03.310154 21290 heartbeater.cc:511] Master 127.2.110.254:36871 requested a full tablet report, sending...
I20260926 06:19:03.310755 21102 ts_manager.cc:194] Registered new tserver with Master: 975696e97c9b474ea0b457fa719ef421 (127.2.110.193:33599)
I20260926 06:19:03.311473 21102 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.110.193:47097
W20260926 06:19:03.392894 21293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:19:03.393162 21293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:19:03.393200 21293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:19:03.394444 21293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:19:03.394526 21293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:19:03.394570 21293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.194
I20260926 06:19:03.395990 21293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:38095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.110.194
--webserver_port=0
--tserver_master_addrs=127.2.110.254:36871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.194
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:19:03.396292 21293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:19:03.396624 21293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:19:03.401827 21301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:03.401907 21293 server_base.cc:1034] running on GCE node
W20260926 06:19:03.401875 21299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:03.401873 21298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:03.402506 21293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:19:03.402954 21293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:19:03.404176 21293 hybrid_clock.cc:648] HybridClock initialized: now 1790403543404150 us; error 46 us; skew 500 ppm
I20260926 06:19:03.406008 21293 webserver.cc:460] Webserver started at http://127.2.110.194:45665/ using document root <none> and password file <none>
I20260926 06:19:03.406328 21293 fs_manager.cc:359] Metadata directory not provided
I20260926 06:19:03.406410 21293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:19:03.406569 21293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:19:03.407984 21293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/ts-1/data/instance:
uuid: "bc7af88c1e3f470dbb8181f0a1d642e7"
format_stamp: "Formatted at 2026-09-26 06:19:03 on dist-test-slave-hndd"
server_key: "dd0622ec059ebe74e9b00c5fcf3875c2"
server_key_iv: "8210bf3b74300f0e47cd9b35fade6f39"
server_key_version: "encryptionkey@0"
I20260926 06:19:03.408486 21293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/ts-1/wal/instance:
uuid: "bc7af88c1e3f470dbb8181f0a1d642e7"
format_stamp: "Formatted at 2026-09-26 06:19:03 on dist-test-slave-hndd"
server_key: "dd0622ec059ebe74e9b00c5fcf3875c2"
server_key_iv: "8210bf3b74300f0e47cd9b35fade6f39"
server_key_version: "encryptionkey@0"
I20260926 06:19:03.410379 21293 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260926 06:19:03.411592 21307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:03.411932 21293 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:19:03.412212 21293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/ts-1/wal
uuid: "bc7af88c1e3f470dbb8181f0a1d642e7"
format_stamp: "Formatted at 2026-09-26 06:19:03 on dist-test-slave-hndd"
server_key: "dd0622ec059ebe74e9b00c5fcf3875c2"
server_key_iv: "8210bf3b74300f0e47cd9b35fade6f39"
server_key_version: "encryptionkey@0"
I20260926 06:19:03.412348 21293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:19:03.426182 21293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:19:03.426533 21293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:19:03.426692 21293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:19:03.426966 21293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:19:03.427309 21293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:19:03.427366 21293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:03.427403 21293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:19:03.427433 21293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:03.433969 21293 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.194:37599
I20260926 06:19:03.434013 21420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.194:37599 every 8 connection(s)
I20260926 06:19:03.434379 21293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
I20260926 06:19:03.440465 21142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:03.435246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:56905 (local address 127.2.110.254:36871)
0926 06:19:03.435358 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:03.435360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:03.435553 (+ 193us) server_negotiation.cc:407] Connection header received
0926 06:19:03.436240 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:03.436242 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:03.436256 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:03.436318 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:03.437059 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:03.437615 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:03.438680 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:03.438949 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:03.439049 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:03.440125 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:03.440127 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:03.440127 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:19:03.440133 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:03.440151 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:19:03.440156 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:03.440217 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:03.440340 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 06:19:03.440372 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 06:19:03.440721 21423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:03.435156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:36871 (local address 127.2.110.194:56905)
0926 06:19:03.435428 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:03.435465 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:03.436089 (+ 624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:03.436407 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:19:03.436410 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:19:03.436419 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:19:03.436933 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:03.436937 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:03.437729 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:03.437730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:03.438539 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:03.438541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:03.439062 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:03.439063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:03.439224 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:03.439933 (+ 709us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:19:03.439934 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:19:03.439964 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:19:03.439990 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:19:03.440223 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:19:03.440230 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:19:03.440233 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:19:03.440330 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 06:19:03.440444 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":90,"threads_started":1}
I20260926 06:19:03.441402 21421 heartbeater.cc:347] Connected to a master server at 127.2.110.254:36871
I20260926 06:19:03.441524 21421 heartbeater.cc:464] Registering TS with master...
I20260926 06:19:03.441736 21421 heartbeater.cc:511] Master 127.2.110.254:36871 requested a full tablet report, sending...
I20260926 06:19:03.442189 21102 ts_manager.cc:194] Registered new tserver with Master: bc7af88c1e3f470dbb8181f0a1d642e7 (127.2.110.194:37599)
I20260926 06:19:03.442723 21102 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.110.194:56905
I20260926 06:19:03.444222 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 21293
I20260926 06:19:03.444343 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/ts-1/wal/instance
I20260926 06:19:03.444615 2491 external_mini_cluster.cc:1428] Setting key f72c08c62fb4945ec39a2675e5125fe8
I20260926 06:19:03.446022 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.195:0
--local_ip_for_outbound_sockets=127.2.110.195
--webserver_interface=127.2.110.195
--webserver_port=0
--tserver_master_addrs=127.2.110.254:36871
--builtin_ntp_servers=127.2.110.212:38095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 06:19:03.536053 21424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:19:03.536326 21424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:19:03.536362 21424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:19:03.537686 21424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:19:03.537757 21424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:19:03.537802 21424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.195
I20260926 06:19:03.539413 21424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:38095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.110.195
--webserver_port=0
--tserver_master_addrs=127.2.110.254:36871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.195
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:19:03.539700 21424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:19:03.540030 21424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:19:03.545091 21430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:03.545403 21429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:03.545368 21424 server_base.cc:1034] running on GCE node
W20260926 06:19:03.545373 21432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:03.545737 21424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:19:03.546025 21424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:19:03.547204 21424 hybrid_clock.cc:648] HybridClock initialized: now 1790403543547168 us; error 52 us; skew 500 ppm
I20260926 06:19:03.548638 21424 webserver.cc:460] Webserver started at http://127.2.110.195:35979/ using document root <none> and password file <none>
I20260926 06:19:03.548902 21424 fs_manager.cc:359] Metadata directory not provided
I20260926 06:19:03.548977 21424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:19:03.549129 21424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:19:03.550324 21424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/ts-2/data/instance:
uuid: "394483d7645747fba3c9d39d3c42acf2"
format_stamp: "Formatted at 2026-09-26 06:19:03 on dist-test-slave-hndd"
server_key: "b1cb3856f7b1e57da28431c3f8b91315"
server_key_iv: "c1987c9c7ce43a2683414cdbfcbf03fa"
server_key_version: "encryptionkey@0"
I20260926 06:19:03.550694 21424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/ts-2/wal/instance:
uuid: "394483d7645747fba3c9d39d3c42acf2"
format_stamp: "Formatted at 2026-09-26 06:19:03 on dist-test-slave-hndd"
server_key: "b1cb3856f7b1e57da28431c3f8b91315"
server_key_iv: "c1987c9c7ce43a2683414cdbfcbf03fa"
server_key_version: "encryptionkey@0"
I20260926 06:19:03.552047 21424 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:19:03.552806 21438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:03.553061 21424 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:19:03.553174 21424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/ts-2/wal
uuid: "394483d7645747fba3c9d39d3c42acf2"
format_stamp: "Formatted at 2026-09-26 06:19:03 on dist-test-slave-hndd"
server_key: "b1cb3856f7b1e57da28431c3f8b91315"
server_key_iv: "c1987c9c7ce43a2683414cdbfcbf03fa"
server_key_version: "encryptionkey@0"
I20260926 06:19:03.553264 21424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:19:03.570463 21424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:19:03.570806 21424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:19:03.570969 21424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:19:03.571213 21424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:19:03.571570 21424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:19:03.571628 21424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:03.571669 21424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:19:03.571698 21424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:03.578539 21424 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.195:44947
I20260926 06:19:03.578581 21551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.195:44947 every 8 connection(s)
I20260926 06:19:03.578934 21424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
I20260926 06:19:03.582726 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 21424
I20260926 06:19:03.582844 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/ts-2/wal/instance
I20260926 06:19:03.583104 2491 external_mini_cluster.cc:1428] Setting key 9be1127cdd9bcf5788ae1be9d293393f
I20260926 06:19:03.584813 21142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:03.579787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:36245 (local address 127.2.110.254:36871)
0926 06:19:03.579865 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:03.579867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:03.580089 (+ 222us) server_negotiation.cc:407] Connection header received
0926 06:19:03.580573 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:03.580574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:03.580585 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:03.580661 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:03.581395 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:03.581903 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:03.583055 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:03.583374 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:03.583446 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:03.584558 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:03.584560 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:03.584560 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:19:03.584566 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:03.584581 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:19:03.584585 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:03.584636 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:03.584707 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 06:19:03.584735 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 06:19:03.585067 21554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:03.579754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:36871 (local address 127.2.110.195:36245)
0926 06:19:03.579953 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:03.579996 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:03.580478 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:03.580713 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:19:03.580717 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:19:03.580724 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:19:03.581223 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:03.581227 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:03.582042 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:03.582043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:03.582917 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:03.582920 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:03.583486 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:03.583488 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:03.583700 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:03.584385 (+ 685us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:19:03.584386 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:19:03.584438 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:19:03.584462 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:19:03.584636 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:19:03.584642 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:19:03.584645 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:19:03.584712 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 06:19:03.584822 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":99,"threads_started":1}
I20260926 06:19:03.585655 21552 heartbeater.cc:347] Connected to a master server at 127.2.110.254:36871
I20260926 06:19:03.585772 21552 heartbeater.cc:464] Registering TS with master...
I20260926 06:19:03.585961 21552 heartbeater.cc:511] Master 127.2.110.254:36871 requested a full tablet report, sending...
I20260926 06:19:03.586357 21102 ts_manager.cc:194] Registered new tserver with Master: 394483d7645747fba3c9d39d3c42acf2 (127.2.110.195:44947)
I20260926 06:19:03.586733 21102 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.110.195:36245
I20260926 06:19:03.595103 2491 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 06:19:03.599670 21142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:03.596529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38744 (local address 127.2.110.254:36871)
0926 06:19:03.596591 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:03.596593 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:03.596693 (+ 100us) server_negotiation.cc:407] Connection header received
0926 06:19:03.596774 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:03.596775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:03.596786 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:03.596854 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:03.597043 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:03.597604 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:03.598383 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:03.598573 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:03.598646 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:03.599372 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:03.599373 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:03.599374 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:19:03.599378 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:03.599394 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:19:03.599398 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:03.599498 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:03.599565 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 06:19:03.599576 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 06:19:03.601399 21102 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38744:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:19:03.603012 21102 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:19:03.609751 21563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:03.605656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37938 (local address 127.2.110.194:37599)
0926 06:19:03.605969 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:03.605974 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:03.605987 (+ 13us) server_negotiation.cc:407] Connection header received
0926 06:19:03.606033 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:03.606035 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:03.606053 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:03.606139 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:19:03.606419 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:03.607676 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:03.608809 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:03.609300 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:03.609430 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:03.609529 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:03.609620 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 06:19:03.609660 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":99,"threads_started":1}
I20260926 06:19:03.609750 21562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:03.605442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.193:33599 (local address 127.0.0.1:42908)
0926 06:19:03.605636 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:03.605646 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:03.605713 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:03.606272 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:19:03.606272 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:19:03.606275 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:19:03.606332 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:03.606334 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:03.607242 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:03.607244 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:03.608403 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:03.608405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:03.609462 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:03.609463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:03.609533 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:03.609538 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 06:19:03.609579 (+ 41us) client_negotiation.cc:240] Negotiation successful
0926 06:19:03.609640 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":89,"threads_started":1}
I20260926 06:19:03.609758 21560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:03.605435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.194:37599 (local address 127.0.0.1:37938)
0926 06:19:03.605601 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:03.605646 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:03.605713 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:03.606225 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:19:03.606229 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:19:03.606231 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:19:03.606328 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:03.606330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:03.607787 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:03.607788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:03.608727 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:03.608729 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:03.609432 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:03.609433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:03.609517 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:03.609530 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 06:19:03.609606 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 06:19:03.609651 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":148,"threads_started":1}
I20260926 06:19:03.610291 21564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:03.605732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42908 (local address 127.2.110.193:33599)
0926 06:19:03.605953 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:03.605957 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:03.605968 (+ 11us) server_negotiation.cc:407] Connection header received
0926 06:19:03.606011 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:03.606013 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:03.606030 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:03.606411 (+ 381us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:19:03.606478 (+ 67us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:03.607153 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:03.608477 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:03.609360 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:03.610006 (+ 646us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:03.610117 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:03.610147 (+ 30us) server_negotiation.cc:299] Negotiation successful
0926 06:19:03.610195 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":112,"threads_started":1}
I20260926 06:19:03.610424 21561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:03.605442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.195:44947 (local address 127.0.0.1:37178)
0926 06:19:03.605949 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:03.605961 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:03.606024 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:03.606328 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:19:03.606329 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:19:03.606331 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:19:03.606370 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:03.606372 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:03.607839 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:03.607840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:03.609084 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:03.609087 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:03.610173 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:03.610174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:03.610253 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:03.610257 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:19:03.610303 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 06:19:03.610353 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"mutex_wait_us":3,"thread_start_us":104,"threads_started":1}
I20260926 06:19:03.611033 21224 tablet_service.cc:1467] Processing CreateTablet for tablet c281d78ffb5745ac95c3d8f9665f15bb (DEFAULT_TABLE table=test-table [id=5d63f676a6474e0eb0ef0f5081b09506]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:19:03.611260 21565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:03.605956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37178 (local address 127.2.110.195:44947)
0926 06:19:03.606151 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:03.606155 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:03.606165 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:19:03.606205 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:03.606207 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:03.606224 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:03.606450 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:19:03.606510 (+ 60us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:03.607713 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:03.609199 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:03.610064 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:03.610991 (+ 927us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:03.611094 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:03.611125 (+ 31us) server_negotiation.cc:299] Negotiation successful
0926 06:19:03.611173 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":103,"threads_started":1}
I20260926 06:19:03.611178 21355 tablet_service.cc:1467] Processing CreateTablet for tablet c281d78ffb5745ac95c3d8f9665f15bb (DEFAULT_TABLE table=test-table [id=5d63f676a6474e0eb0ef0f5081b09506]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:19:03.611434 21224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c281d78ffb5745ac95c3d8f9665f15bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:19:03.611542 21355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c281d78ffb5745ac95c3d8f9665f15bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:19:03.612032 21486 tablet_service.cc:1467] Processing CreateTablet for tablet c281d78ffb5745ac95c3d8f9665f15bb (DEFAULT_TABLE table=test-table [id=5d63f676a6474e0eb0ef0f5081b09506]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:19:03.612392 21486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c281d78ffb5745ac95c3d8f9665f15bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:19:03.615018 21566 tablet_bootstrap.cc:492] T c281d78ffb5745ac95c3d8f9665f15bb P 975696e97c9b474ea0b457fa719ef421: Bootstrap starting.
I20260926 06:19:03.615361 21567 tablet_bootstrap.cc:492] T c281d78ffb5745ac95c3d8f9665f15bb P bc7af88c1e3f470dbb8181f0a1d642e7: Bootstrap starting.
I20260926 06:19:03.615818 21566 tablet_bootstrap.cc:654] T c281d78ffb5745ac95c3d8f9665f15bb P 975696e97c9b474ea0b457fa719ef421: Neither blocks nor log segments found. Creating new log.
I20260926 06:19:03.616050 21567 tablet_bootstrap.cc:654] T c281d78ffb5745ac95c3d8f9665f15bb P bc7af88c1e3f470dbb8181f0a1d642e7: Neither blocks nor log segments found. Creating new log.
I20260926 06:19:03.616236 21566 log.cc:824] T c281d78ffb5745ac95c3d8f9665f15bb P 975696e97c9b474ea0b457fa719ef421: Log is configured to *not* fsync() on all Append() calls
I20260926 06:19:03.616277 21568 tablet_bootstrap.cc:492] T c281d78ffb5745ac95c3d8f9665f15bb P 394483d7645747fba3c9d39d3c42acf2: Bootstrap starting.
I20260926 06:19:03.616406 21567 log.cc:824] T c281d78ffb5745ac95c3d8f9665f15bb P bc7af88c1e3f470dbb8181f0a1d642e7: Log is configured to *not* fsync() on all Append() calls
I20260926 06:19:03.617069 21566 tablet_bootstrap.cc:492] T c281d78ffb5745ac95c3d8f9665f15bb P 975696e97c9b474ea0b457fa719ef421: No bootstrap required, opened a new log
I20260926 06:19:03.617142 21566 ts_tablet_manager.cc:1397] T c281d78ffb5745ac95c3d8f9665f15bb P 975696e97c9b474ea0b457fa719ef421: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:19:03.617208 21568 tablet_bootstrap.cc:654] T c281d78ffb5745ac95c3d8f9665f15bb P 394483d7645747fba3c9d39d3c42acf2: Neither blocks nor log segments found. Creating new log.
I20260926 06:19:03.617620 21568 log.cc:824] T c281d78ffb5745ac95c3d8f9665f15bb P 394483d7645747fba3c9d39d3c42acf2: Log is configured to *not* fsync() on all Append() calls
I20260926 06:19:03.617676 21567 tablet_bootstrap.cc:492] T c281d78ffb5745ac95c3d8f9665f15bb P bc7af88c1e3f470dbb8181f0a1d642e7: No bootstrap required, opened a new log
I20260926 06:19:03.617764 21567 ts_tablet_manager.cc:1397] T c281d78ffb5745ac95c3d8f9665f15bb P bc7af88c1e3f470dbb8181f0a1d642e7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:19:03.618496 21568 tablet_bootstrap.cc:492] T c281d78ffb5745ac95c3d8f9665f15bb P 394483d7645747fba3c9d39d3c42acf2: No bootstrap required, opened a new log
I20260926 06:19:03.618593 21568 ts_tablet_manager.cc:1397] T c281d78ffb5745ac95c3d8f9665f15bb P 394483d7645747fba3c9d39d3c42acf2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 06:19:03.620088 21566 raft_consensus.cc:348] T c281d78ffb5745ac95c3d8f9665f15bb P 975696e97c9b474ea0b457fa719ef421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "975696e97c9b474ea0b457fa719ef421" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 33599 } } peers { permanent_uuid: "bc7af88c1e3f470dbb8181f0a1d642e7" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 37599 } } peers { permanent_uuid: "394483d7645747fba3c9d39d3c42acf2" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 44947 } }
I20260926 06:19:03.620293 21566 raft_consensus.cc:374] T c281d78ffb5745ac95c3d8f9665f15bb P 975696e97c9b474ea0b457fa719ef421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:19:03.620333 21566 raft_consensus.cc:729] T c281d78ffb5745ac95c3d8f9665f15bb P 975696e97c9b474ea0b457fa719ef421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975696e97c9b474ea0b457fa719ef421, State: Initialized, Role: FOLLOWER
I20260926 06:19:03.620422 21568 raft_consensus.cc:348] T c281d78ffb5745ac95c3d8f9665f15bb P 394483d7645747fba3c9d39d3c42acf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "975696e97c9b474ea0b457fa719ef421" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 33599 } } peers { permanent_uuid: "bc7af88c1e3f470dbb8181f0a1d642e7" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 37599 } } peers { permanent_uuid: "394483d7645747fba3c9d39d3c42acf2" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 44947 } }
I20260926 06:19:03.620498 21566 consensus_queue.cc:260] T c281d78ffb5745ac95c3d8f9665f15bb P 975696e97c9b474ea0b457fa719ef421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "975696e97c9b474ea0b457fa719ef421" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 33599 } } peers { permanent_uuid: "bc7af88c1e3f470dbb8181f0a1d642e7" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 37599 } } peers { permanent_uuid: "394483d7645747fba3c9d39d3c42acf2" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 44947 } }
I20260926 06:19:03.620594 21568 raft_consensus.cc:374] T c281d78ffb5745ac95c3d8f9665f15bb P 394483d7645747fba3c9d39d3c42acf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:19:03.620517 21567 raft_consensus.cc:348] T c281d78ffb5745ac95c3d8f9665f15bb P bc7af88c1e3f470dbb8181f0a1d642e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "975696e97c9b474ea0b457fa719ef421" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 33599 } } peers { permanent_uuid: "bc7af88c1e3f470dbb8181f0a1d642e7" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 37599 } } peers { permanent_uuid: "394483d7645747fba3c9d39d3c42acf2" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 44947 } }
I20260926 06:19:03.620656 21568 raft_consensus.cc:729] T c281d78ffb5745ac95c3d8f9665f15bb P 394483d7645747fba3c9d39d3c42acf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 394483d7645747fba3c9d39d3c42acf2, State: Initialized, Role: FOLLOWER
I20260926 06:19:03.620690 21567 raft_consensus.cc:374] T c281d78ffb5745ac95c3d8f9665f15bb P bc7af88c1e3f470dbb8181f0a1d642e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:19:03.620744 21567 raft_consensus.cc:729] T c281d78ffb5745ac95c3d8f9665f15bb P bc7af88c1e3f470dbb8181f0a1d642e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc7af88c1e3f470dbb8181f0a1d642e7, State: Initialized, Role: FOLLOWER
I20260926 06:19:03.620787 21566 ts_tablet_manager.cc:1428] T c281d78ffb5745ac95c3d8f9665f15bb P 975696e97c9b474ea0b457fa719ef421: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260926 06:19:03.620795 21568 consensus_queue.cc:260] T c281d78ffb5745ac95c3d8f9665f15bb P 394483d7645747fba3c9d39d3c42acf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "975696e97c9b474ea0b457fa719ef421" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 33599 } } peers { permanent_uuid: "bc7af88c1e3f470dbb8181f0a1d642e7" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 37599 } } peers { permanent_uuid: "394483d7645747fba3c9d39d3c42acf2" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 44947 } }
I20260926 06:19:03.620921 21567 consensus_queue.cc:260] T c281d78ffb5745ac95c3d8f9665f15bb P bc7af88c1e3f470dbb8181f0a1d642e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "975696e97c9b474ea0b457fa719ef421" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 33599 } } peers { permanent_uuid: "bc7af88c1e3f470dbb8181f0a1d642e7" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 37599 } } peers { permanent_uuid: "394483d7645747fba3c9d39d3c42acf2" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 44947 } }
I20260926 06:19:03.621007 21290 heartbeater.cc:503] Master 127.2.110.254:36871 was elected leader, sending a full tablet report...
I20260926 06:19:03.621047 21568 ts_tablet_manager.cc:1428] T c281d78ffb5745ac95c3d8f9665f15bb P 394483d7645747fba3c9d39d3c42acf2: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260926 06:19:03.621061 21552 heartbeater.cc:503] Master 127.2.110.254:36871 was elected leader, sending a full tablet report...
I20260926 06:19:03.621218 21567 ts_tablet_manager.cc:1428] T c281d78ffb5745ac95c3d8f9665f15bb P bc7af88c1e3f470dbb8181f0a1d642e7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 06:19:03.621326 21421 heartbeater.cc:503] Master 127.2.110.254:36871 was elected leader, sending a full tablet report...
W20260926 06:19:03.685050 21422 tablet.cc:2369] T c281d78ffb5745ac95c3d8f9665f15bb P bc7af88c1e3f470dbb8181f0a1d642e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 06:19:03.803844 21291 tablet.cc:2369] T c281d78ffb5745ac95c3d8f9665f15bb P 975696e97c9b474ea0b457fa719ef421: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:19:03.805604 21573 raft_consensus.cc:482] T c281d78ffb5745ac95c3d8f9665f15bb P 394483d7645747fba3c9d39d3c42acf2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:19:03.805787 21573 raft_consensus.cc:504] T c281d78ffb5745ac95c3d8f9665f15bb P 394483d7645747fba3c9d39d3c42acf2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "975696e97c9b474ea0b457fa719ef421" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 33599 } } peers { permanent_uuid: "bc7af88c1e3f470dbb8181f0a1d642e7" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 37599 } } peers { permanent_uuid: "394483d7645747fba3c9d39d3c42acf2" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 44947 } }
I20260926 06:19:03.806171 21573 leader_election.cc:288] T c281d78ffb5745ac95c3d8f9665f15bb P 394483d7645747fba3c9d39d3c42acf2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 975696e97c9b474ea0b457fa719ef421 (127.2.110.193:33599), bc7af88c1e3f470dbb8181f0a1d642e7 (127.2.110.194:37599)
I20260926 06:19:03.809914 21564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:03.806628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:52471 (local address 127.2.110.193:33599)
0926 06:19:03.806677 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:03.806679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:03.806694 (+ 15us) server_negotiation.cc:407] Connection header received
0926 06:19:03.806789 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:03.806790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:03.806803 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:03.806861 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:19:03.807049 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:03.807735 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:03.809026 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:03.809500 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:03.809598 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:03.809607 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:03.809789 (+ 182us) server_negotiation.cc:299] Negotiation successful
0926 06:19:03.809845 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 06:19:03.809891 21554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:03.806487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.193:33599 (local address 127.2.110.195:52471)
0926 06:19:03.806593 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:03.806608 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:03.806676 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:03.806878 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:19:03.806879 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:19:03.806882 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:19:03.806966 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:03.806968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:03.807932 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:03.807935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:03.808895 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:03.808899 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:03.809598 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:03.809599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:03.809684 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:03.809688 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:19:03.809742 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 06:19:03.809801 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"mutex_wait_us":1}
I20260926 06:19:03.809895 21575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:03.806486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.194:37599 (local address 127.2.110.195:42361)
0926 06:19:03.806682 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:03.806692 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:03.806765 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:03.806945 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:19:03.806946 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:19:03.806947 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:19:03.807027 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:03.807028 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:03.807980 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:03.807981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:03.808911 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:03.808914 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:03.809660 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:03.809660 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:03.809734 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:03.809738 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:19:03.809793 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 06:19:03.809820 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":17,"thread_start_us":106,"threads_started":1}
I20260926 06:19:03.809973 21563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:03.806609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:42361 (local address 127.2.110.194:37599)
0926 06:19:03.806654 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:03.806656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:03.806775 (+ 119us) server_negotiation.cc:407] Connection header received
0926 06:19:03.806817 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:03.806817 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:03.806827 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:03.806920 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:19:03.807144 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:03.807787 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:03.809066 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:03.809530 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:03.809679 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:03.809701 (+ 22us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:03.809818 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 06:19:03.809892 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 06:19:03.810330 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c281d78ffb5745ac95c3d8f9665f15bb" candidate_uuid: "394483d7645747fba3c9d39d3c42acf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975696e97c9b474ea0b457fa719ef421" is_pre_election: true
I20260926 06:19:03.810329 21375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c281d78ffb5745ac95c3d8f9665f15bb" candidate_uuid: "394483d7645747fba3c9d39d3c42acf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc7af88c1e3f470dbb8181f0a1d642e7" is_pre_election: true
I20260926 06:19:03.810529 21244 raft_consensus.cc:2454] T c281d78ffb5745ac95c3d8f9665f15bb P 975696e97c9b474ea0b457fa719ef421 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 394483d7645747fba3c9d39d3c42acf2 in term 0.
I20260926 06:19:03.810534 21375 raft_consensus.cc:2454] T c281d78ffb5745ac95c3d8f9665f15bb P bc7af88c1e3f470dbb8181f0a1d642e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 394483d7645747fba3c9d39d3c42acf2 in term 0.
I20260926 06:19:03.810814 21442 leader_election.cc:302] T c281d78ffb5745ac95c3d8f9665f15bb P 394483d7645747fba3c9d39d3c42acf2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 394483d7645747fba3c9d39d3c42acf2, bc7af88c1e3f470dbb8181f0a1d642e7; no voters:
I20260926 06:19:03.810968 21573 raft_consensus.cc:2781] T c281d78ffb5745ac95c3d8f9665f15bb P 394483d7645747fba3c9d39d3c42acf2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:19:03.811025 21573 raft_consensus.cc:482] T c281d78ffb5745ac95c3d8f9665f15bb P 394483d7645747fba3c9d39d3c42acf2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:19:03.811053 21573 raft_consensus.cc:3037] T c281d78ffb5745ac95c3d8f9665f15bb P 394483d7645747fba3c9d39d3c42acf2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:19:03.811751 21573 raft_consensus.cc:504] T c281d78ffb5745ac95c3d8f9665f15bb P 394483d7645747fba3c9d39d3c42acf2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "975696e97c9b474ea0b457fa719ef421" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 33599 } } peers { permanent_uuid: "bc7af88c1e3f470dbb8181f0a1d642e7" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 37599 } } peers { permanent_uuid: "394483d7645747fba3c9d39d3c42acf2" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 44947 } }
I20260926 06:19:03.811936 21573 leader_election.cc:288] T c281d78ffb5745ac95c3d8f9665f15bb P 394483d7645747fba3c9d39d3c42acf2 [CANDIDATE]: Term 1 election: Requested vote from peers 975696e97c9b474ea0b457fa719ef421 (127.2.110.193:33599), bc7af88c1e3f470dbb8181f0a1d642e7 (127.2.110.194:37599)
I20260926 06:19:03.812095 21375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c281d78ffb5745ac95c3d8f9665f15bb" candidate_uuid: "394483d7645747fba3c9d39d3c42acf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc7af88c1e3f470dbb8181f0a1d642e7"
I20260926 06:19:03.812180 21375 raft_consensus.cc:3037] T c281d78ffb5745ac95c3d8f9665f15bb P bc7af88c1e3f470dbb8181f0a1d642e7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:19:03.812103 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c281d78ffb5745ac95c3d8f9665f15bb" candidate_uuid: "394483d7645747fba3c9d39d3c42acf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975696e97c9b474ea0b457fa719ef421"
I20260926 06:19:03.812332 21244 raft_consensus.cc:3037] T c281d78ffb5745ac95c3d8f9665f15bb P 975696e97c9b474ea0b457fa719ef421 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:19:03.812987 21375 raft_consensus.cc:2454] T c281d78ffb5745ac95c3d8f9665f15bb P bc7af88c1e3f470dbb8181f0a1d642e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 394483d7645747fba3c9d39d3c42acf2 in term 1.
I20260926 06:19:03.813174 21442 leader_election.cc:302] T c281d78ffb5745ac95c3d8f9665f15bb P 394483d7645747fba3c9d39d3c42acf2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 394483d7645747fba3c9d39d3c42acf2, bc7af88c1e3f470dbb8181f0a1d642e7; no voters:
I20260926 06:19:03.813284 21573 raft_consensus.cc:2781] T c281d78ffb5745ac95c3d8f9665f15bb P 394483d7645747fba3c9d39d3c42acf2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:19:03.813462 21573 raft_consensus.cc:686] T c281d78ffb5745ac95c3d8f9665f15bb P 394483d7645747fba3c9d39d3c42acf2 [term 1 LEADER]: Becoming Leader. State: Replica: 394483d7645747fba3c9d39d3c42acf2, State: Running, Role: LEADER
I20260926 06:19:03.813606 21244 raft_consensus.cc:2454] T c281d78ffb5745ac95c3d8f9665f15bb P 975696e97c9b474ea0b457fa719ef421 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 394483d7645747fba3c9d39d3c42acf2 in term 1.
I20260926 06:19:03.813575 21573 consensus_queue.cc:237] T c281d78ffb5745ac95c3d8f9665f15bb P 394483d7645747fba3c9d39d3c42acf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "975696e97c9b474ea0b457fa719ef421" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 33599 } } peers { permanent_uuid: "bc7af88c1e3f470dbb8181f0a1d642e7" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 37599 } } peers { permanent_uuid: "394483d7645747fba3c9d39d3c42acf2" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 44947 } }
I20260926 06:19:03.814536 21102 catalog_manager.cc:5495] T c281d78ffb5745ac95c3d8f9665f15bb P 394483d7645747fba3c9d39d3c42acf2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 394483d7645747fba3c9d39d3c42acf2 (127.2.110.195). New cstate: current_term: 1 leader_uuid: "394483d7645747fba3c9d39d3c42acf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "975696e97c9b474ea0b457fa719ef421" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 33599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc7af88c1e3f470dbb8181f0a1d642e7" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 37599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "394483d7645747fba3c9d39d3c42acf2" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 44947 } health_report { overall_health: HEALTHY } } }
W20260926 06:19:03.829622 21553 tablet.cc:2369] T c281d78ffb5745ac95c3d8f9665f15bb P 394483d7645747fba3c9d39d3c42acf2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:19:03.838737 21565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:03.836079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37190 (local address 127.2.110.195:44947)
0926 06:19:03.836165 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:03.836167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:03.836175 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:19:03.836220 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:03.836221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:03.836248 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:03.836349 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:19:03.836505 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:03.836977 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:03.837833 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:03.838103 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:03.838179 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:03.838337 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:19:03.838440 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:19:03.838553 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:03.838607 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 06:19:03.838619 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 06:19:03.840837 21375 raft_consensus.cc:1261] T c281d78ffb5745ac95c3d8f9665f15bb P bc7af88c1e3f470dbb8181f0a1d642e7 [term 1 FOLLOWER]: Refusing update from remote peer 394483d7645747fba3c9d39d3c42acf2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:19:03.840962 21244 raft_consensus.cc:1261] T c281d78ffb5745ac95c3d8f9665f15bb P 975696e97c9b474ea0b457fa719ef421 [term 1 FOLLOWER]: Refusing update from remote peer 394483d7645747fba3c9d39d3c42acf2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:19:03.841430 21573 consensus_queue.cc:1035] T c281d78ffb5745ac95c3d8f9665f15bb P 394483d7645747fba3c9d39d3c42acf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc7af88c1e3f470dbb8181f0a1d642e7" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 37599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:19:03.841624 21576 consensus_queue.cc:1035] T c281d78ffb5745ac95c3d8f9665f15bb P 394483d7645747fba3c9d39d3c42acf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975696e97c9b474ea0b457fa719ef421" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 33599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:19:03.844071 21579 mvcc.cc:205] Tried to move back new op lower bound from 7333492915568955392 to 7333492915461193728. Current Snapshot: MvccSnapshot[applied={T|T < 7333492915568955392}]
I20260926 06:19:03.844501 21581 mvcc.cc:205] Tried to move back new op lower bound from 7333492915568955392 to 7333492915461193728. Current Snapshot: MvccSnapshot[applied={T|T < 7333492915568955392}]
I20260926 06:19:03.844678 21580 mvcc.cc:205] Tried to move back new op lower bound from 7333492915568955392 to 7333492915461193728. Current Snapshot: MvccSnapshot[applied={T|T < 7333492915568955392}]
I20260926 06:19:03.847698 21099 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:19:03.847813 21099 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:19:03.848592 21099 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d591889470a342648ea5f73d810a7754: Sending DeleteTablet for 3 replicas of tablet c281d78ffb5745ac95c3d8f9665f15bb
I20260926 06:19:03.848930 21486 tablet_service.cc:1514] Processing DeleteTablet for tablet c281d78ffb5745ac95c3d8f9665f15bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:19:03 UTC) from {username='slave'} at 127.0.0.1:37178
I20260926 06:19:03.849037 21355 tablet_service.cc:1514] Processing DeleteTablet for tablet c281d78ffb5745ac95c3d8f9665f15bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:19:03 UTC) from {username='slave'} at 127.0.0.1:37938
I20260926 06:19:03.849021 21224 tablet_service.cc:1514] Processing DeleteTablet for tablet c281d78ffb5745ac95c3d8f9665f15bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:19:03 UTC) from {username='slave'} at 127.0.0.1:42908
I20260926 06:19:03.849227 21588 tablet_replica.cc:330] T c281d78ffb5745ac95c3d8f9665f15bb P 394483d7645747fba3c9d39d3c42acf2: stopping tablet replica
I20260926 06:19:03.849283 21590 tablet_replica.cc:330] T c281d78ffb5745ac95c3d8f9665f15bb P bc7af88c1e3f470dbb8181f0a1d642e7: stopping tablet replica
I20260926 06:19:03.849423 21590 raft_consensus.cc:2229] T c281d78ffb5745ac95c3d8f9665f15bb P bc7af88c1e3f470dbb8181f0a1d642e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:19:03.849426 21588 raft_consensus.cc:2229] T c281d78ffb5745ac95c3d8f9665f15bb P 394483d7645747fba3c9d39d3c42acf2 [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:19:03.849552 21590 raft_consensus.cc:2258] T c281d78ffb5745ac95c3d8f9665f15bb P bc7af88c1e3f470dbb8181f0a1d642e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:19:03.849555 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 21162
I20260926 06:19:03.849668 21588 raft_consensus.cc:2258] T c281d78ffb5745ac95c3d8f9665f15bb P 394483d7645747fba3c9d39d3c42acf2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:19:03.850500 21590 ts_tablet_manager.cc:1905] T c281d78ffb5745ac95c3d8f9665f15bb P bc7af88c1e3f470dbb8181f0a1d642e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:19:03.850795 21588 ts_tablet_manager.cc:1905] T c281d78ffb5745ac95c3d8f9665f15bb P 394483d7645747fba3c9d39d3c42acf2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:19:03.852838 21588 ts_tablet_manager.cc:1918] T c281d78ffb5745ac95c3d8f9665f15bb P 394483d7645747fba3c9d39d3c42acf2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:19:03.852929 21588 log.cc:1196] T c281d78ffb5745ac95c3d8f9665f15bb P 394483d7645747fba3c9d39d3c42acf2: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/ts-2/wal/wals/c281d78ffb5745ac95c3d8f9665f15bb
I20260926 06:19:03.852862 21590 ts_tablet_manager.cc:1918] T c281d78ffb5745ac95c3d8f9665f15bb P bc7af88c1e3f470dbb8181f0a1d642e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:19:03.853104 21590 log.cc:1196] T c281d78ffb5745ac95c3d8f9665f15bb P bc7af88c1e3f470dbb8181f0a1d642e7: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403292369990-2491-0/minicluster-data/ts-1/wal/wals/c281d78ffb5745ac95c3d8f9665f15bb
I20260926 06:19:03.853248 21588 ts_tablet_manager.cc:1939] T c281d78ffb5745ac95c3d8f9665f15bb P 394483d7645747fba3c9d39d3c42acf2: Deleting consensus metadata
I20260926 06:19:03.853503 21590 ts_tablet_manager.cc:1939] T c281d78ffb5745ac95c3d8f9665f15bb P bc7af88c1e3f470dbb8181f0a1d642e7: Deleting consensus metadata
I20260926 06:19:03.853756 21087 catalog_manager.cc:4841] TS 394483d7645747fba3c9d39d3c42acf2 (127.2.110.195:44947): tablet c281d78ffb5745ac95c3d8f9665f15bb (table test-table [id=5d63f676a6474e0eb0ef0f5081b09506]) successfully deleted
I20260926 06:19:03.854173 21089 catalog_manager.cc:4841] TS bc7af88c1e3f470dbb8181f0a1d642e7 (127.2.110.194:37599): tablet c281d78ffb5745ac95c3d8f9665f15bb (table test-table [id=5d63f676a6474e0eb0ef0f5081b09506]) successfully deleted
W20260926 06:19:03.856371 21086 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.110.193:33599 (error 108)
I20260926 06:19:03.856667 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 21293
I20260926 06:19:03.857383 21561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:03.856766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.193:33599 (local address 127.0.0.1:42922)
0926 06:19:03.857244 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:03.857296 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.193:33599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453}
W20260926 06:19:03.857535 21086 catalog_manager.cc:4584] TS 975696e97c9b474ea0b457fa719ef421 (127.2.110.193:33599): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c281d78ffb5745ac95c3d8f9665f15bb: Network error: Client connection negotiation failed: client connection to 127.2.110.193:33599: connect: Connection refused (error 111)
I20260926 06:19:03.863929 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 21424
I20260926 06:19:03.869556 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 21071
2026-09-26T06:19:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (888 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:19:03 dist-test-slave-hndd krb5kdc[21594](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:19:03 dist-test-slave-hndd krb5kdc[21594](info): set up 2 sockets
Sep 26 06:19:03 dist-test-slave-hndd krb5kdc[21594](info): commencing operation
krb5kdc: starting...
W20260926 06:19:04.969303 2491 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Sep 26 06:19:05 dist-test-slave-hndd krb5kdc[21594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:19:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:19:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
I20260926 06:19:05.101711 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:33133
--webserver_interface=127.2.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:42161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.110.254:33133
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:19:05.187393 21607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:19:05.187652 21607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:19:05.187693 21607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:19:05.188953 21607 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:19:05.189028 21607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:19:05.189055 21607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:19:05.189087 21607 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:19:05.189112 21607 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:19:05.190864 21607 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:42161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.110.254:33133
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:33133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:19:05.191161 21607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:19:05.191469 21607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 06:19:05.197381 21607 server_base.cc:1034] running on GCE node
W20260926 06:19:05.197322 21612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:05.197357 21613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:05.197322 21615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:05.197799 21607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:19:05.198135 21607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:19:05.199313 21607 hybrid_clock.cc:648] HybridClock initialized: now 1790403545199295 us; error 42 us; skew 500 ppm
Sep 26 06:19:05 dist-test-slave-hndd krb5kdc[21594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403545, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:19:05.201936 21607 init.cc:377] Logged in from keytab as kudu/127.2.110.254@KRBTEST.COM (short username kudu)
I20260926 06:19:05.202358 21607 webserver.cc:460] Webserver started at http://127.2.110.254:42347/ using document root <none> and password file <none>
I20260926 06:19:05.202566 21607 fs_manager.cc:359] Metadata directory not provided
I20260926 06:19:05.202608 21607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:19:05.202698 21607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:19:05.203747 21607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/master-0/data/instance:
uuid: "0faaf5951fcd4d3eade415852054ec80"
format_stamp: "Formatted at 2026-09-26 06:19:05 on dist-test-slave-hndd"
server_key: "452c42b128f8f78a1159684fdad82a8a"
server_key_iv: "570a111c36b6fe1b5e8e99740e072749"
server_key_version: "encryptionkey@0"
I20260926 06:19:05.204058 21607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/master-0/wal/instance:
uuid: "0faaf5951fcd4d3eade415852054ec80"
format_stamp: "Formatted at 2026-09-26 06:19:05 on dist-test-slave-hndd"
server_key: "452c42b128f8f78a1159684fdad82a8a"
server_key_iv: "570a111c36b6fe1b5e8e99740e072749"
server_key_version: "encryptionkey@0"
I20260926 06:19:05.205428 21607 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 06:19:05.206252 21622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:05.206430 21607 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:19:05.206508 21607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "0faaf5951fcd4d3eade415852054ec80"
format_stamp: "Formatted at 2026-09-26 06:19:05 on dist-test-slave-hndd"
server_key: "452c42b128f8f78a1159684fdad82a8a"
server_key_iv: "570a111c36b6fe1b5e8e99740e072749"
server_key_version: "encryptionkey@0"
I20260926 06:19:05.206573 21607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:19:05.224196 21607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:19:05.226476 21607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:19:05.226605 21607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:19:05.230619 21607 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.254:33133
I20260926 06:19:05.230664 21674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.254:33133 every 8 connection(s)
I20260926 06:19:05.231025 21607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:19:05.232131 21675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:19:05.234488 21675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0faaf5951fcd4d3eade415852054ec80: Bootstrap starting.
I20260926 06:19:05.235090 21675 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0faaf5951fcd4d3eade415852054ec80: Neither blocks nor log segments found. Creating new log.
I20260926 06:19:05.235388 21675 log.cc:824] T 00000000000000000000000000000000 P 0faaf5951fcd4d3eade415852054ec80: Log is configured to *not* fsync() on all Append() calls
I20260926 06:19:05.236223 21675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0faaf5951fcd4d3eade415852054ec80: No bootstrap required, opened a new log
I20260926 06:19:05.237861 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 21607
I20260926 06:19:05.237987 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/master-0/wal/instance
I20260926 06:19:05.238166 2491 external_mini_cluster.cc:1428] Setting key 6f06689b02d2dda03b734265f0f200a0
I20260926 06:19:05.238076 21675 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0faaf5951fcd4d3eade415852054ec80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0faaf5951fcd4d3eade415852054ec80" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 33133 } }
I20260926 06:19:05.238245 21675 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0faaf5951fcd4d3eade415852054ec80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:19:05.238293 21675 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0faaf5951fcd4d3eade415852054ec80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0faaf5951fcd4d3eade415852054ec80, State: Initialized, Role: FOLLOWER
I20260926 06:19:05.238462 21675 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0faaf5951fcd4d3eade415852054ec80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0faaf5951fcd4d3eade415852054ec80" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 33133 } }
I20260926 06:19:05.238551 21675 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0faaf5951fcd4d3eade415852054ec80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:19:05.238610 21675 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0faaf5951fcd4d3eade415852054ec80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:19:05.238667 21675 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0faaf5951fcd4d3eade415852054ec80 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:19:05.239248 21675 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0faaf5951fcd4d3eade415852054ec80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0faaf5951fcd4d3eade415852054ec80" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 33133 } }
I20260926 06:19:05.239369 21675 leader_election.cc:302] T 00000000000000000000000000000000 P 0faaf5951fcd4d3eade415852054ec80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0faaf5951fcd4d3eade415852054ec80; no voters:
I20260926 06:19:05.239564 21675 leader_election.cc:288] T 00000000000000000000000000000000 P 0faaf5951fcd4d3eade415852054ec80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:19:05.239621 21680 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0faaf5951fcd4d3eade415852054ec80 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:19:05.239786 21680 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0faaf5951fcd4d3eade415852054ec80 [term 1 LEADER]: Becoming Leader. State: Replica: 0faaf5951fcd4d3eade415852054ec80, State: Running, Role: LEADER
I20260926 06:19:05.239845 21675 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0faaf5951fcd4d3eade415852054ec80 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:19:05.239923 21680 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0faaf5951fcd4d3eade415852054ec80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0faaf5951fcd4d3eade415852054ec80" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 33133 } }
I20260926 06:19:05.240300 21680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0faaf5951fcd4d3eade415852054ec80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0faaf5951fcd4d3eade415852054ec80. Latest consensus state: current_term: 1 leader_uuid: "0faaf5951fcd4d3eade415852054ec80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0faaf5951fcd4d3eade415852054ec80" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 33133 } } }
I20260926 06:19:05.240442 21680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0faaf5951fcd4d3eade415852054ec80 [sys.catalog]: This master's current role is: LEADER
I20260926 06:19:05.240675 21688 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:19:05.241094 21682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0faaf5951fcd4d3eade415852054ec80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0faaf5951fcd4d3eade415852054ec80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0faaf5951fcd4d3eade415852054ec80" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 33133 } } }
I20260926 06:19:05.241175 21688 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:19:05.242484 21682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0faaf5951fcd4d3eade415852054ec80 [sys.catalog]: This master's current role is: LEADER
I20260926 06:19:05.243544 21688 catalog_manager.cc:1347] Generated new cluster ID: 269f1f755da1419195e87932b4a54a1a
I20260926 06:19:05.243603 21688 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 06:19:05 dist-test-slave-hndd krb5kdc[21594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:19:05.250602 21688 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:19:05.250758 21679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:05.238656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37790 (local address 127.2.110.254:33133)
0926 06:19:05.238838 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:05.238847 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:05.238890 (+ 43us) server_negotiation.cc:407] Connection header received
0926 06:19:05.239235 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:05.239241 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:05.239294 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:05.239465 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:05.240280 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:05.241923 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:05.242896 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:05.243841 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:05.243953 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:05.246804 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:05.246809 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:05.246809 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:19:05.246822 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:05.248912 (+ 2090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:05.249325 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:05.249328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:05.249328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:05.249493 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:05.249714 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:05.249714 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:05.249715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:05.250038 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:05.250205 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:05.250334 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 06:19:05.250486 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":128,"thread_start_us":94,"threads_started":1}
I20260926 06:19:05.251237 21688 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:19:05.259302 21688 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0faaf5951fcd4d3eade415852054ec80: Generated new TSK 0
I20260926 06:19:05.259505 21688 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab.
Entry for principal kudu/127.2.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab.
I20260926 06:19:05.289291 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.193:0
--local_ip_for_outbound_sockets=127.2.110.193
--webserver_interface=127.2.110.193
--webserver_port=0
--tserver_master_addrs=127.2.110.254:33133
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab
--principal=kudu/127.2.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:42161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:19:05.375133 21701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:19:05.375802 21701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:19:05.375898 21701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:19:05.378347 21701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:19:05.378464 21701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:19:05.378530 21701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.193
I20260926 06:19:05.380678 21701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:42161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.193.keytab
--principal=kudu/127.2.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.110.193
--webserver_port=0
--tserver_master_addrs=127.2.110.254:33133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.193
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:19:05.381002 21701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:19:05.381466 21701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:19:05.387432 21706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:05.387436 21707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:05.387436 21709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:05.387882 21701 server_base.cc:1034] running on GCE node
I20260926 06:19:05.388099 21701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:19:05.388360 21701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:19:05.389544 21701 hybrid_clock.cc:648] HybridClock initialized: now 1790403545389532 us; error 38 us; skew 500 ppm
Sep 26 06:19:05 dist-test-slave-hndd krb5kdc[21594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403545, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:19:05.392212 21701 init.cc:377] Logged in from keytab as kudu/127.2.110.193@KRBTEST.COM (short username kudu)
I20260926 06:19:05.392709 21701 webserver.cc:460] Webserver started at http://127.2.110.193:37611/ using document root <none> and password file <none>
I20260926 06:19:05.392958 21701 fs_manager.cc:359] Metadata directory not provided
I20260926 06:19:05.393030 21701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:19:05.393182 21701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:19:05.394577 21701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/ts-0/data/instance:
uuid: "59a4d18422bf4313accfebe3c72c6387"
format_stamp: "Formatted at 2026-09-26 06:19:05 on dist-test-slave-hndd"
server_key: "ea5444b4c89b4f4188078360f5fd95e2"
server_key_iv: "d1289089484d77b82ae1b7b6cc5c034a"
server_key_version: "encryptionkey@0"
I20260926 06:19:05.395045 21701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/ts-0/wal/instance:
uuid: "59a4d18422bf4313accfebe3c72c6387"
format_stamp: "Formatted at 2026-09-26 06:19:05 on dist-test-slave-hndd"
server_key: "ea5444b4c89b4f4188078360f5fd95e2"
server_key_iv: "d1289089484d77b82ae1b7b6cc5c034a"
server_key_version: "encryptionkey@0"
I20260926 06:19:05.396508 21701 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:19:05.397291 21716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:05.397555 21701 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:19:05.397656 21701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/ts-0/wal
uuid: "59a4d18422bf4313accfebe3c72c6387"
format_stamp: "Formatted at 2026-09-26 06:19:05 on dist-test-slave-hndd"
server_key: "ea5444b4c89b4f4188078360f5fd95e2"
server_key_iv: "d1289089484d77b82ae1b7b6cc5c034a"
server_key_version: "encryptionkey@0"
I20260926 06:19:05.397746 21701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:19:05.417305 21701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:19:05.419435 21701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:19:05.419627 21701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:19:05.419888 21701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:19:05.420217 21701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:19:05.420272 21701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:05.420317 21701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:19:05.420341 21701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:05.426272 21701 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.193:39175
I20260926 06:19:05.426332 21829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.193:39175 every 8 connection(s)
I20260926 06:19:05.426697 21701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
Sep 26 06:19:05 dist-test-slave-hndd krb5kdc[21594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403545, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.193@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:19:05.435844 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 21701
I20260926 06:19:05.436033 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/ts-0/wal/instance
I20260926 06:19:05.436376 2491 external_mini_cluster.cc:1428] Setting key c07e6e9ee2b1656ba22da94adfd7bfc8
I20260926 06:19:05.439787 21679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:05.427836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:44983 (local address 127.2.110.254:33133)
0926 06:19:05.427887 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:05.427889 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:05.428707 (+ 818us) server_negotiation.cc:407] Connection header received
0926 06:19:05.429319 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:05.429320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:05.429392 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:05.429476 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:05.431012 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:05.431685 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:05.432940 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:05.433153 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:05.433233 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:05.435981 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:05.435983 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:05.435984 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:19:05.435993 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:05.437979 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:05.438547 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:05.438548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:05.438549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:05.438620 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:05.438936 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:05.438937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:05.438937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:05.439050 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:05.439105 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:05.439577 (+ 472us) server_negotiation.cc:299] Negotiation successful
0926 06:19:05.439662 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 06:19:05.439911 21832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:05.427803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:33133 (local address 127.2.110.193:44983)
0926 06:19:05.428564 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:05.428607 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:05.429152 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:05.429549 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:19:05.429553 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:19:05.430496 (+ 943us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:19:05.430831 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:05.430842 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:05.431848 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:05.431850 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:05.432809 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:05.432812 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:05.433266 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:05.433267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:05.433394 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:05.433931 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:19:05.433940 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:19:05.435802 (+ 1862us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:19:05.438072 (+ 2270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:19:05.438081 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:19:05.438084 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:19:05.438447 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:19:05.438700 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:19:05.438701 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:19:05.438702 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:19:05.438865 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:19:05.439117 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:19:05.439124 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:19:05.439319 (+ 195us) client_negotiation.cc:769] Sending connection context
0926 06:19:05.439447 (+ 128us) client_negotiation.cc:240] Negotiation successful
0926 06:19:05.439646 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"thread_start_us":98,"threads_started":1}
I20260926 06:19:05.440505 21830 heartbeater.cc:347] Connected to a master server at 127.2.110.254:33133
I20260926 06:19:05.440622 21830 heartbeater.cc:464] Registering TS with master...
I20260926 06:19:05.440835 21830 heartbeater.cc:511] Master 127.2.110.254:33133 requested a full tablet report, sending...
I20260926 06:19:05.441440 21639 ts_manager.cc:194] Registered new tserver with Master: 59a4d18422bf4313accfebe3c72c6387 (127.2.110.193:39175)
I20260926 06:19:05.442409 21639 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.193@KRBTEST.COM'} at 127.2.110.193:44983
WARNING: no policy specified for kudu/127.2.110.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab.
Entry for principal kudu/127.2.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab.
I20260926 06:19:05.465416 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.194:0
--local_ip_for_outbound_sockets=127.2.110.194
--webserver_interface=127.2.110.194
--webserver_port=0
--tserver_master_addrs=127.2.110.254:33133
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab
--principal=kudu/127.2.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:42161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:19:05.556072 21835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:19:05.556300 21835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:19:05.556334 21835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:19:05.557621 21835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:19:05.557672 21835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:19:05.557701 21835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.194
I20260926 06:19:05.559209 21835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:42161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.194.keytab
--principal=kudu/127.2.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.110.194
--webserver_port=0
--tserver_master_addrs=127.2.110.254:33133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.194
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:19:05.559486 21835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:19:05.559734 21835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:19:05.565198 21843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:05.565434 21835 server_base.cc:1034] running on GCE node
W20260926 06:19:05.565146 21841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:05.565429 21840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:05.565871 21835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:19:05.566151 21835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:19:05.567405 21835 hybrid_clock.cc:648] HybridClock initialized: now 1790403545567383 us; error 38 us; skew 500 ppm
Sep 26 06:19:05 dist-test-slave-hndd krb5kdc[21594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403545, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:19:05.570281 21835 init.cc:377] Logged in from keytab as kudu/127.2.110.194@KRBTEST.COM (short username kudu)
I20260926 06:19:05.570837 21835 webserver.cc:460] Webserver started at http://127.2.110.194:33881/ using document root <none> and password file <none>
I20260926 06:19:05.571077 21835 fs_manager.cc:359] Metadata directory not provided
I20260926 06:19:05.571144 21835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:19:05.571305 21835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:19:05.572422 21835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/ts-1/data/instance:
uuid: "723b2b9b57114b568c9fb840f98b38a5"
format_stamp: "Formatted at 2026-09-26 06:19:05 on dist-test-slave-hndd"
server_key: "a17a30dd5d21a4a56723a059fdb2b507"
server_key_iv: "dda68177834b45b6f2bf84685ef352b0"
server_key_version: "encryptionkey@0"
I20260926 06:19:05.572819 21835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/ts-1/wal/instance:
uuid: "723b2b9b57114b568c9fb840f98b38a5"
format_stamp: "Formatted at 2026-09-26 06:19:05 on dist-test-slave-hndd"
server_key: "a17a30dd5d21a4a56723a059fdb2b507"
server_key_iv: "dda68177834b45b6f2bf84685ef352b0"
server_key_version: "encryptionkey@0"
I20260926 06:19:05.574719 21835 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260926 06:19:05.575775 21850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:05.576103 21835 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:19:05.576251 21835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/ts-1/wal
uuid: "723b2b9b57114b568c9fb840f98b38a5"
format_stamp: "Formatted at 2026-09-26 06:19:05 on dist-test-slave-hndd"
server_key: "a17a30dd5d21a4a56723a059fdb2b507"
server_key_iv: "dda68177834b45b6f2bf84685ef352b0"
server_key_version: "encryptionkey@0"
I20260926 06:19:05.576342 21835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:19:05.590533 21835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:19:05.592734 21835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:19:05.592917 21835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:19:05.593241 21835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:19:05.593621 21835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:19:05.593657 21835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:05.593680 21835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:19:05.593729 21835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:05.600404 21835 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.194:33787
I20260926 06:19:05.600446 21963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.194:33787 every 8 connection(s)
I20260926 06:19:05.600838 21835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
I20260926 06:19:05.601310 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 21835
I20260926 06:19:05.601506 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/ts-1/wal/instance
I20260926 06:19:05.601847 2491 external_mini_cluster.cc:1428] Setting key 8b501af7770b8e8f4d098a73d7989f2d
Sep 26 06:19:05 dist-test-slave-hndd krb5kdc[21594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403545, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.194@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.110.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.195@KRBTEST.COM" created.
I20260926 06:19:05.614101 21679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:05.601674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:43989 (local address 127.2.110.254:33133)
0926 06:19:05.601731 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:05.601734 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:05.601977 (+ 243us) server_negotiation.cc:407] Connection header received
0926 06:19:05.602574 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:05.602575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:05.602587 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:05.602646 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:05.603732 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:05.604481 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:05.606171 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:05.606521 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:05.606611 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:05.609769 (+ 3158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:05.609772 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:05.609772 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:19:05.609782 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:05.611774 (+ 1992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:05.612444 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:05.612446 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:05.612446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:05.612525 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:05.613023 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:05.613026 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:05.613027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:05.613302 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:05.613501 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:05.613889 (+ 388us) server_negotiation.cc:299] Negotiation successful
0926 06:19:05.613979 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 06:19:05.614287 21966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:05.601583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:33133 (local address 127.2.110.194:43989)
0926 06:19:05.601811 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:05.601875 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:05.602461 (+ 586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:05.602730 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:19:05.602734 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:19:05.603420 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:19:05.603625 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:05.603629 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:05.604675 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:05.604677 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:05.606049 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:05.606055 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:05.606616 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:05.606617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:05.606745 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:05.607375 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:19:05.607383 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:19:05.609649 (+ 2266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:19:05.611942 (+ 2293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:19:05.611954 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:19:05.611958 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:19:05.612313 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:19:05.612662 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:19:05.612663 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:19:05.612664 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:19:05.612912 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:19:05.613486 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:19:05.613493 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:19:05.613643 (+ 150us) client_negotiation.cc:769] Sending connection context
0926 06:19:05.613758 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 06:19:05.613977 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":82,"threads_started":1}
I20260926 06:19:05.614910 21964 heartbeater.cc:347] Connected to a master server at 127.2.110.254:33133
I20260926 06:19:05.615088 21964 heartbeater.cc:464] Registering TS with master...
I20260926 06:19:05.615319 21964 heartbeater.cc:511] Master 127.2.110.254:33133 requested a full tablet report, sending...
I20260926 06:19:05.615783 21639 ts_manager.cc:194] Registered new tserver with Master: 723b2b9b57114b568c9fb840f98b38a5 (127.2.110.194:33787)
I20260926 06:19:05.616196 21639 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.194@KRBTEST.COM'} at 127.2.110.194:43989
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab.
Entry for principal kudu/127.2.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab.
I20260926 06:19:05.633234 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.195:0
--local_ip_for_outbound_sockets=127.2.110.195
--webserver_interface=127.2.110.195
--webserver_port=0
--tserver_master_addrs=127.2.110.254:33133
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab
--principal=kudu/127.2.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:42161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:19:05.716401 21969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:19:05.716675 21969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:19:05.716717 21969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:19:05.718073 21969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:19:05.718151 21969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:19:05.718200 21969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.195
I20260926 06:19:05.719740 21969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:42161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.195.keytab
--principal=kudu/127.2.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.110.195
--webserver_port=0
--tserver_master_addrs=127.2.110.254:33133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.195
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:19:05.720032 21969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:19:05.720287 21969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:19:05.725735 21975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:05.725713 21977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:05.725713 21974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:05.726226 21969 server_base.cc:1034] running on GCE node
I20260926 06:19:05.726454 21969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:19:05.726732 21969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:19:05.727905 21969 hybrid_clock.cc:648] HybridClock initialized: now 1790403545727878 us; error 47 us; skew 500 ppm
Sep 26 06:19:05 dist-test-slave-hndd krb5kdc[21594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403545, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:19:05.730808 21969 init.cc:377] Logged in from keytab as kudu/127.2.110.195@KRBTEST.COM (short username kudu)
I20260926 06:19:05.731331 21969 webserver.cc:460] Webserver started at http://127.2.110.195:33357/ using document root <none> and password file <none>
I20260926 06:19:05.731576 21969 fs_manager.cc:359] Metadata directory not provided
I20260926 06:19:05.731650 21969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:19:05.731802 21969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:19:05.732914 21969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/ts-2/data/instance:
uuid: "fc4fc74e57f64094acfcecf84b8d94b8"
format_stamp: "Formatted at 2026-09-26 06:19:05 on dist-test-slave-hndd"
server_key: "966518ddc3d43765d74097f5524e92c3"
server_key_iv: "00230344011e4b00e879c642445f7d65"
server_key_version: "encryptionkey@0"
I20260926 06:19:05.733299 21969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/ts-2/wal/instance:
uuid: "fc4fc74e57f64094acfcecf84b8d94b8"
format_stamp: "Formatted at 2026-09-26 06:19:05 on dist-test-slave-hndd"
server_key: "966518ddc3d43765d74097f5524e92c3"
server_key_iv: "00230344011e4b00e879c642445f7d65"
server_key_version: "encryptionkey@0"
I20260926 06:19:05.734760 21969 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:19:05.735671 21984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:05.735906 21969 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:19:05.736021 21969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/ts-2/wal
uuid: "fc4fc74e57f64094acfcecf84b8d94b8"
format_stamp: "Formatted at 2026-09-26 06:19:05 on dist-test-slave-hndd"
server_key: "966518ddc3d43765d74097f5524e92c3"
server_key_iv: "00230344011e4b00e879c642445f7d65"
server_key_version: "encryptionkey@0"
I20260926 06:19:05.736104 21969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:19:05.757174 21969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:19:05.759820 21969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:19:05.759965 21969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:19:05.760155 21969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:19:05.760444 21969 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:19:05.760479 21969 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:05.760504 21969 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:19:05.760516 21969 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:05.766906 21969 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.195:45783
I20260926 06:19:05.766938 22097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.195:45783 every 8 connection(s)
I20260926 06:19:05.767313 21969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
I20260926 06:19:05.769047 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 21969
I20260926 06:19:05.769304 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/ts-2/wal/instance
I20260926 06:19:05.769649 2491 external_mini_cluster.cc:1428] Setting key bc4f32f7e9fe1d4ffd6abddf7864b8e9
Sep 26 06:19:05 dist-test-slave-hndd krb5kdc[21594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403545, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.195@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:19:05.779882 21679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:05.768647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:59799 (local address 127.2.110.254:33133)
0926 06:19:05.768793 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:05.768796 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:05.769110 (+ 314us) server_negotiation.cc:407] Connection header received
0926 06:19:05.770104 (+ 994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:05.770105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:05.770118 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:05.770201 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:05.771423 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:05.771911 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:05.773218 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:05.773491 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:05.773550 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:05.776235 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:05.776237 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:05.776237 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:19:05.776244 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:05.777958 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:05.778622 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:05.778625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:05.778625 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:05.778760 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:05.779063 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:05.779064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:05.779064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:05.779220 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:05.779285 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:05.779614 (+ 329us) server_negotiation.cc:299] Negotiation successful
0926 06:19:05.779686 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260926 06:19:05.779983 22100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:05.768616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:33133 (local address 127.2.110.195:59799)
0926 06:19:05.768861 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:05.768938 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:05.769921 (+ 983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:05.770335 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:19:05.770340 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:19:05.771112 (+ 772us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:19:05.771331 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:05.771337 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:05.772089 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:05.772090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:05.773108 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:05.773111 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:05.773585 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:05.773586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:05.773692 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:05.774339 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:19:05.774353 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:19:05.776091 (+ 1738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:19:05.778056 (+ 1965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:19:05.778064 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:19:05.778066 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:19:05.778347 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:19:05.778863 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:19:05.778863 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:19:05.778863 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:19:05.778984 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:19:05.779298 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:19:05.779303 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:19:05.779469 (+ 166us) client_negotiation.cc:769] Sending connection context
0926 06:19:05.779582 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 06:19:05.779725 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":149,"threads_started":1}
I20260926 06:19:05.780519 22098 heartbeater.cc:347] Connected to a master server at 127.2.110.254:33133
I20260926 06:19:05.780645 22098 heartbeater.cc:464] Registering TS with master...
I20260926 06:19:05.780875 22098 heartbeater.cc:511] Master 127.2.110.254:33133 requested a full tablet report, sending...
I20260926 06:19:05.781409 21639 ts_manager.cc:194] Registered new tserver with Master: fc4fc74e57f64094acfcecf84b8d94b8 (127.2.110.195:45783)
I20260926 06:19:05.781805 2491 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 06:19:05.781826 21639 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.110.195@KRBTEST.COM'} at 127.2.110.195:59799
I20260926 06:19:05.790686 21679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:05.783137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37794 (local address 127.2.110.254:33133)
0926 06:19:05.783360 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:05.783363 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:05.783383 (+ 20us) server_negotiation.cc:407] Connection header received
0926 06:19:05.783424 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:05.783425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:05.783438 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:05.783530 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:05.784174 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:05.784734 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:05.785839 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:05.786056 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:05.786123 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:05.787477 (+ 1354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:05.787479 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:05.787479 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:19:05.787485 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:05.788920 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:05.789418 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:05.789421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:05.789422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:05.789607 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:05.789915 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:05.789916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:05.789917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:05.790254 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:05.790354 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:05.790533 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 06:19:05.790612 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260926 06:19:05.792645 21639 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37794:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:19:05.794507 21639 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:19:05.802158 22113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:05.797322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.194:33787 (local address 127.0.0.1:37820)
0926 06:19:05.797707 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:05.797718 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:05.797778 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:05.798116 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:19:05.798117 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:19:05.798119 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:19:05.798181 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:05.798182 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:05.799580 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:05.799581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:05.800779 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:05.800781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:05.801825 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:05.801826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:05.801920 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:05.801937 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 06:19:05.802016 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 06:19:05.802070 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":344,"threads_started":1}
I20260926 06:19:05.802328 22114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:05.797761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37820 (local address 127.2.110.194:33787)
0926 06:19:05.797931 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:05.797934 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:05.797942 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:19:05.797999 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:05.798000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:05.798016 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:05.798258 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:19:05.798336 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:05.799457 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:05.800889 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:05.801739 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:05.801825 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:05.801888 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:05.802100 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 06:19:05.802136 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":93,"threads_started":1}
I20260926 06:19:05.802634 22110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:05.797189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.195:45783 (local address 127.0.0.1:50856)
0926 06:19:05.797510 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:05.797556 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:05.797637 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:05.797995 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:19:05.797999 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:19:05.798003 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:19:05.798095 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:05.798098 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:05.799179 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:05.799180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:05.800644 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:05.800648 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:05.802392 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:05.802393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:05.802469 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:05.802473 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:19:05.802523 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 06:19:05.802566 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":123,"threads_started":1}
I20260926 06:19:05.803094 22111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:05.797418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50856 (local address 127.2.110.195:45783)
0926 06:19:05.797616 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:05.797622 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:05.797638 (+ 16us) server_negotiation.cc:407] Connection header received
0926 06:19:05.797793 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:05.797797 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:05.797817 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:05.798182 (+ 365us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:19:05.798252 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:05.799056 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:05.800746 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:05.801734 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:05.801828 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:05.801889 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:05.802785 (+ 896us) server_negotiation.cc:299] Negotiation successful
0926 06:19:05.803007 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":124,"threads_started":1}
I20260926 06:19:05.803071 21898 tablet_service.cc:1467] Processing CreateTablet for tablet 2e29721cfa5f4dc7a63eb75b58bfadf2 (DEFAULT_TABLE table=test-table [id=fc8922bce7794ac3aa209852956a9b60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:19:05.803369 22109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:05.797322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.193:39175 (local address 127.0.0.1:36296)
0926 06:19:05.797715 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:05.797724 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:05.797778 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:05.798212 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:19:05.798213 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:19:05.798215 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:19:05.798280 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:05.798283 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:05.800854 (+ 2571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:05.800855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:05.801818 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:05.801821 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:05.802994 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:05.802995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:05.803055 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:05.803060 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 06:19:05.803122 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 06:19:05.803163 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":3,"thread_start_us":90,"threads_started":1}
I20260926 06:19:05.803541 21898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e29721cfa5f4dc7a63eb75b58bfadf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:19:05.803606 22112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:05.797553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36296 (local address 127.2.110.193:39175)
0926 06:19:05.797980 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:05.797985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:05.797995 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:19:05.798033 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:05.798035 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:05.798050 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:05.798132 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:19:05.798375 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:05.799460 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:05.801922 (+ 2462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:05.802663 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:05.802731 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:05.802790 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:05.803471 (+ 681us) server_negotiation.cc:299] Negotiation successful
0926 06:19:05.803528 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":92,"threads_started":1}
I20260926 06:19:05.803862 22032 tablet_service.cc:1467] Processing CreateTablet for tablet 2e29721cfa5f4dc7a63eb75b58bfadf2 (DEFAULT_TABLE table=test-table [id=fc8922bce7794ac3aa209852956a9b60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:19:05.804679 22032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e29721cfa5f4dc7a63eb75b58bfadf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:19:05.808200 22115 tablet_bootstrap.cc:492] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P fc4fc74e57f64094acfcecf84b8d94b8: Bootstrap starting.
I20260926 06:19:05.809213 21764 tablet_service.cc:1467] Processing CreateTablet for tablet 2e29721cfa5f4dc7a63eb75b58bfadf2 (DEFAULT_TABLE table=test-table [id=fc8922bce7794ac3aa209852956a9b60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:19:05.809628 21764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e29721cfa5f4dc7a63eb75b58bfadf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:19:05.809989 22115 tablet_bootstrap.cc:654] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P fc4fc74e57f64094acfcecf84b8d94b8: Neither blocks nor log segments found. Creating new log.
I20260926 06:19:05.810815 22115 log.cc:824] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P fc4fc74e57f64094acfcecf84b8d94b8: Log is configured to *not* fsync() on all Append() calls
I20260926 06:19:05.811223 22116 tablet_bootstrap.cc:492] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P 723b2b9b57114b568c9fb840f98b38a5: Bootstrap starting.
I20260926 06:19:05.811650 22115 tablet_bootstrap.cc:492] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P fc4fc74e57f64094acfcecf84b8d94b8: No bootstrap required, opened a new log
I20260926 06:19:05.811750 22115 ts_tablet_manager.cc:1397] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P fc4fc74e57f64094acfcecf84b8d94b8: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 06:19:05.812467 22116 tablet_bootstrap.cc:654] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P 723b2b9b57114b568c9fb840f98b38a5: Neither blocks nor log segments found. Creating new log.
I20260926 06:19:05.813076 22118 tablet_bootstrap.cc:492] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P 59a4d18422bf4313accfebe3c72c6387: Bootstrap starting.
I20260926 06:19:05.813235 22116 log.cc:824] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P 723b2b9b57114b568c9fb840f98b38a5: Log is configured to *not* fsync() on all Append() calls
I20260926 06:19:05.813969 22116 tablet_bootstrap.cc:492] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P 723b2b9b57114b568c9fb840f98b38a5: No bootstrap required, opened a new log
I20260926 06:19:05.814059 22116 ts_tablet_manager.cc:1397] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P 723b2b9b57114b568c9fb840f98b38a5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 06:19:05.814116 22118 tablet_bootstrap.cc:654] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P 59a4d18422bf4313accfebe3c72c6387: Neither blocks nor log segments found. Creating new log.
I20260926 06:19:05.814675 22115 raft_consensus.cc:348] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P fc4fc74e57f64094acfcecf84b8d94b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc4fc74e57f64094acfcecf84b8d94b8" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 45783 } } peers { permanent_uuid: "723b2b9b57114b568c9fb840f98b38a5" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 33787 } } peers { permanent_uuid: "59a4d18422bf4313accfebe3c72c6387" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 39175 } }
I20260926 06:19:05.814936 22115 raft_consensus.cc:374] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P fc4fc74e57f64094acfcecf84b8d94b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:19:05.814980 22115 raft_consensus.cc:729] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P fc4fc74e57f64094acfcecf84b8d94b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc4fc74e57f64094acfcecf84b8d94b8, State: Initialized, Role: FOLLOWER
I20260926 06:19:05.815006 22118 log.cc:824] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P 59a4d18422bf4313accfebe3c72c6387: Log is configured to *not* fsync() on all Append() calls
I20260926 06:19:05.815156 22115 consensus_queue.cc:260] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P fc4fc74e57f64094acfcecf84b8d94b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc4fc74e57f64094acfcecf84b8d94b8" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 45783 } } peers { permanent_uuid: "723b2b9b57114b568c9fb840f98b38a5" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 33787 } } peers { permanent_uuid: "59a4d18422bf4313accfebe3c72c6387" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 39175 } }
I20260926 06:19:05.815493 22115 ts_tablet_manager.cc:1428] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P fc4fc74e57f64094acfcecf84b8d94b8: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260926 06:19:05.815629 22098 heartbeater.cc:503] Master 127.2.110.254:33133 was elected leader, sending a full tablet report...
I20260926 06:19:05.815822 22118 tablet_bootstrap.cc:492] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P 59a4d18422bf4313accfebe3c72c6387: No bootstrap required, opened a new log
I20260926 06:19:05.815899 22118 ts_tablet_manager.cc:1397] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P 59a4d18422bf4313accfebe3c72c6387: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260926 06:19:05.816404 22116 raft_consensus.cc:348] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P 723b2b9b57114b568c9fb840f98b38a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc4fc74e57f64094acfcecf84b8d94b8" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 45783 } } peers { permanent_uuid: "723b2b9b57114b568c9fb840f98b38a5" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 33787 } } peers { permanent_uuid: "59a4d18422bf4313accfebe3c72c6387" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 39175 } }
I20260926 06:19:05.816601 22116 raft_consensus.cc:374] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P 723b2b9b57114b568c9fb840f98b38a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:19:05.816637 22116 raft_consensus.cc:729] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P 723b2b9b57114b568c9fb840f98b38a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 723b2b9b57114b568c9fb840f98b38a5, State: Initialized, Role: FOLLOWER
I20260926 06:19:05.816757 22116 consensus_queue.cc:260] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P 723b2b9b57114b568c9fb840f98b38a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc4fc74e57f64094acfcecf84b8d94b8" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 45783 } } peers { permanent_uuid: "723b2b9b57114b568c9fb840f98b38a5" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 33787 } } peers { permanent_uuid: "59a4d18422bf4313accfebe3c72c6387" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 39175 } }
I20260926 06:19:05.817039 22116 ts_tablet_manager.cc:1428] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P 723b2b9b57114b568c9fb840f98b38a5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 06:19:05.817071 21964 heartbeater.cc:503] Master 127.2.110.254:33133 was elected leader, sending a full tablet report...
I20260926 06:19:05.818034 22118 raft_consensus.cc:348] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P 59a4d18422bf4313accfebe3c72c6387 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc4fc74e57f64094acfcecf84b8d94b8" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 45783 } } peers { permanent_uuid: "723b2b9b57114b568c9fb840f98b38a5" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 33787 } } peers { permanent_uuid: "59a4d18422bf4313accfebe3c72c6387" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 39175 } }
I20260926 06:19:05.818251 22118 raft_consensus.cc:374] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P 59a4d18422bf4313accfebe3c72c6387 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:19:05.818305 22118 raft_consensus.cc:729] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P 59a4d18422bf4313accfebe3c72c6387 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59a4d18422bf4313accfebe3c72c6387, State: Initialized, Role: FOLLOWER
I20260926 06:19:05.818423 22118 consensus_queue.cc:260] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P 59a4d18422bf4313accfebe3c72c6387 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc4fc74e57f64094acfcecf84b8d94b8" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 45783 } } peers { permanent_uuid: "723b2b9b57114b568c9fb840f98b38a5" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 33787 } } peers { permanent_uuid: "59a4d18422bf4313accfebe3c72c6387" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 39175 } }
I20260926 06:19:05.818694 22118 ts_tablet_manager.cc:1428] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P 59a4d18422bf4313accfebe3c72c6387: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 06:19:05.818747 21830 heartbeater.cc:503] Master 127.2.110.254:33133 was elected leader, sending a full tablet report...
W20260926 06:19:05.851541 21965 tablet.cc:2369] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P 723b2b9b57114b568c9fb840f98b38a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:19:05.859988 22123 raft_consensus.cc:482] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P 59a4d18422bf4313accfebe3c72c6387 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:19:05.860147 22123 raft_consensus.cc:504] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P 59a4d18422bf4313accfebe3c72c6387 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc4fc74e57f64094acfcecf84b8d94b8" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 45783 } } peers { permanent_uuid: "723b2b9b57114b568c9fb840f98b38a5" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 33787 } } peers { permanent_uuid: "59a4d18422bf4313accfebe3c72c6387" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 39175 } }
I20260926 06:19:05.860587 22123 leader_election.cc:288] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P 59a4d18422bf4313accfebe3c72c6387 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc4fc74e57f64094acfcecf84b8d94b8 (127.2.110.195:45783), 723b2b9b57114b568c9fb840f98b38a5 (127.2.110.194:33787)
I20260926 06:19:05.864465 22111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:05.860993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:56049 (local address 127.2.110.195:45783)
0926 06:19:05.861044 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:05.861046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:05.861055 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:19:05.861086 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:05.861087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:05.861097 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:05.861302 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:19:05.861518 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:05.862126 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:05.863512 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:05.864053 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:05.864134 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:05.864144 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:05.864340 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 06:19:05.864392 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 06:19:05.864465 21832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:05.860854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.195:45783 (local address 127.2.110.193:56049)
0926 06:19:05.860932 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:05.860945 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:05.860995 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:05.861222 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:19:05.861223 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:19:05.861225 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:19:05.861386 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:05.861389 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:05.862253 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:05.862254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:05.863434 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:05.863436 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:05.864161 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:05.864162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:05.864250 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:05.864254 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:19:05.864310 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 06:19:05.864349 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 06:19:05.864760 22114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:05.860979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:49709 (local address 127.2.110.194:33787)
0926 06:19:05.861035 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:05.861037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:05.861179 (+ 142us) server_negotiation.cc:407] Connection header received
0926 06:19:05.861479 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:05.861481 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:05.861499 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:05.861582 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:19:05.862001 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:05.862518 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:05.863746 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:05.864286 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:05.864388 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:05.864395 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:05.864612 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 06:19:05.864665 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 06:19:05.864908 22052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e29721cfa5f4dc7a63eb75b58bfadf2" candidate_uuid: "59a4d18422bf4313accfebe3c72c6387" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc4fc74e57f64094acfcecf84b8d94b8" is_pre_election: true
I20260926 06:19:05.864982 22124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:05.860855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.194:33787 (local address 127.2.110.193:49709)
0926 06:19:05.861101 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:05.861109 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:05.861152 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:05.861682 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:19:05.861684 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:19:05.861688 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:19:05.861889 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:05.861894 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:05.862622 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:05.862623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:05.863667 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:05.863669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:05.864436 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:05.864437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:05.864510 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:05.864514 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:19:05.864860 (+ 346us) client_negotiation.cc:240] Negotiation successful
0926 06:19:05.864913 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":50,"thread_start_us":119,"threads_started":1}
I20260926 06:19:05.865094 22052 raft_consensus.cc:2454] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P fc4fc74e57f64094acfcecf84b8d94b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59a4d18422bf4313accfebe3c72c6387 in term 0.
I20260926 06:19:05.865396 21719 leader_election.cc:302] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P 59a4d18422bf4313accfebe3c72c6387 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59a4d18422bf4313accfebe3c72c6387, fc4fc74e57f64094acfcecf84b8d94b8; no voters:
I20260926 06:19:05.865691 22123 raft_consensus.cc:2781] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P 59a4d18422bf4313accfebe3c72c6387 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:19:05.865797 22123 raft_consensus.cc:482] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P 59a4d18422bf4313accfebe3c72c6387 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:19:05.865832 22123 raft_consensus.cc:3037] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P 59a4d18422bf4313accfebe3c72c6387 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:19:05.865810 21918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e29721cfa5f4dc7a63eb75b58bfadf2" candidate_uuid: "59a4d18422bf4313accfebe3c72c6387" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "723b2b9b57114b568c9fb840f98b38a5" is_pre_election: true
I20260926 06:19:05.865952 21918 raft_consensus.cc:2454] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P 723b2b9b57114b568c9fb840f98b38a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59a4d18422bf4313accfebe3c72c6387 in term 0.
I20260926 06:19:05.866755 22123 raft_consensus.cc:504] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P 59a4d18422bf4313accfebe3c72c6387 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc4fc74e57f64094acfcecf84b8d94b8" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 45783 } } peers { permanent_uuid: "723b2b9b57114b568c9fb840f98b38a5" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 33787 } } peers { permanent_uuid: "59a4d18422bf4313accfebe3c72c6387" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 39175 } }
I20260926 06:19:05.866925 22123 leader_election.cc:288] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P 59a4d18422bf4313accfebe3c72c6387 [CANDIDATE]: Term 1 election: Requested vote from peers fc4fc74e57f64094acfcecf84b8d94b8 (127.2.110.195:45783), 723b2b9b57114b568c9fb840f98b38a5 (127.2.110.194:33787)
I20260926 06:19:05.867101 21918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e29721cfa5f4dc7a63eb75b58bfadf2" candidate_uuid: "59a4d18422bf4313accfebe3c72c6387" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "723b2b9b57114b568c9fb840f98b38a5"
I20260926 06:19:05.867154 22052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e29721cfa5f4dc7a63eb75b58bfadf2" candidate_uuid: "59a4d18422bf4313accfebe3c72c6387" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc4fc74e57f64094acfcecf84b8d94b8"
I20260926 06:19:05.867285 21918 raft_consensus.cc:3037] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P 723b2b9b57114b568c9fb840f98b38a5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:19:05.867408 22052 raft_consensus.cc:3037] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P fc4fc74e57f64094acfcecf84b8d94b8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:19:05.868527 21918 raft_consensus.cc:2454] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P 723b2b9b57114b568c9fb840f98b38a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59a4d18422bf4313accfebe3c72c6387 in term 1.
I20260926 06:19:05.868527 22052 raft_consensus.cc:2454] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P fc4fc74e57f64094acfcecf84b8d94b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59a4d18422bf4313accfebe3c72c6387 in term 1.
I20260926 06:19:05.868813 21720 leader_election.cc:302] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P 59a4d18422bf4313accfebe3c72c6387 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59a4d18422bf4313accfebe3c72c6387, 723b2b9b57114b568c9fb840f98b38a5; no voters:
I20260926 06:19:05.868921 22123 raft_consensus.cc:2781] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P 59a4d18422bf4313accfebe3c72c6387 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:19:05.869118 22123 raft_consensus.cc:686] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P 59a4d18422bf4313accfebe3c72c6387 [term 1 LEADER]: Becoming Leader. State: Replica: 59a4d18422bf4313accfebe3c72c6387, State: Running, Role: LEADER
I20260926 06:19:05.869401 22123 consensus_queue.cc:237] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P 59a4d18422bf4313accfebe3c72c6387 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc4fc74e57f64094acfcecf84b8d94b8" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 45783 } } peers { permanent_uuid: "723b2b9b57114b568c9fb840f98b38a5" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 33787 } } peers { permanent_uuid: "59a4d18422bf4313accfebe3c72c6387" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 39175 } }
I20260926 06:19:05.870747 21638 catalog_manager.cc:5495] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P 59a4d18422bf4313accfebe3c72c6387 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59a4d18422bf4313accfebe3c72c6387 (127.2.110.193). New cstate: current_term: 1 leader_uuid: "59a4d18422bf4313accfebe3c72c6387" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc4fc74e57f64094acfcecf84b8d94b8" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 45783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "723b2b9b57114b568c9fb840f98b38a5" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 33787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59a4d18422bf4313accfebe3c72c6387" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 39175 } health_report { overall_health: HEALTHY } } }
I20260926 06:19:05.884037 22112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:05.881171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36310 (local address 127.2.110.193:39175)
0926 06:19:05.881297 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:05.881299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:05.881311 (+ 12us) server_negotiation.cc:407] Connection header received
0926 06:19:05.881410 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:05.881412 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:05.881433 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:05.881496 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:19:05.881641 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:05.882172 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:05.882953 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:05.883202 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:05.883267 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:05.883581 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:19:05.883689 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:19:05.883811 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:05.883949 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 06:19:05.883971 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260926 06:19:05.886184 22052 raft_consensus.cc:1261] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P fc4fc74e57f64094acfcecf84b8d94b8 [term 1 FOLLOWER]: Refusing update from remote peer 59a4d18422bf4313accfebe3c72c6387: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:19:05.886178 21918 raft_consensus.cc:1261] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P 723b2b9b57114b568c9fb840f98b38a5 [term 1 FOLLOWER]: Refusing update from remote peer 59a4d18422bf4313accfebe3c72c6387: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:19:05.886554 22123 consensus_queue.cc:1035] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P 59a4d18422bf4313accfebe3c72c6387 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc4fc74e57f64094acfcecf84b8d94b8" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 45783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:19:05.886708 22125 consensus_queue.cc:1035] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P 59a4d18422bf4313accfebe3c72c6387 [LEADER]: Connected to new peer: Peer: permanent_uuid: "723b2b9b57114b568c9fb840f98b38a5" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 33787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:19:05.889885 22128 mvcc.cc:205] Tried to move back new op lower bound from 7333492923946778624 to 7333492923882283008. Current Snapshot: MvccSnapshot[applied={T|T < 7333492923946778624}]
I20260926 06:19:05.893361 21639 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:19:05.893488 21639 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:19:05.894366 21639 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0faaf5951fcd4d3eade415852054ec80: Sending DeleteTablet for 3 replicas of tablet 2e29721cfa5f4dc7a63eb75b58bfadf2
I20260926 06:19:05.894768 22032 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e29721cfa5f4dc7a63eb75b58bfadf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:19:05 UTC) from {username='kudu', principal='kudu/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:50856
I20260926 06:19:05.894809 21764 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e29721cfa5f4dc7a63eb75b58bfadf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:19:05 UTC) from {username='kudu', principal='kudu/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:36296
I20260926 06:19:05.894775 21898 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e29721cfa5f4dc7a63eb75b58bfadf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:19:05 UTC) from {username='kudu', principal='kudu/127.2.110.254@KRBTEST.COM'} at 127.0.0.1:37820
I20260926 06:19:05.895030 22138 tablet_replica.cc:330] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P fc4fc74e57f64094acfcecf84b8d94b8: stopping tablet replica
I20260926 06:19:05.895133 22138 raft_consensus.cc:2229] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P fc4fc74e57f64094acfcecf84b8d94b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:19:05.895300 22139 tablet_replica.cc:330] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P 59a4d18422bf4313accfebe3c72c6387: stopping tablet replica
I20260926 06:19:05.895394 22139 raft_consensus.cc:2229] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P 59a4d18422bf4313accfebe3c72c6387 [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:19:05.895397 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 21701
I20260926 06:19:05.895691 22139 raft_consensus.cc:2258] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P 59a4d18422bf4313accfebe3c72c6387 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:19:05.895884 22140 tablet_replica.cc:330] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P 723b2b9b57114b568c9fb840f98b38a5: stopping tablet replica
I20260926 06:19:05.895982 22140 raft_consensus.cc:2229] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P 723b2b9b57114b568c9fb840f98b38a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:19:05.896287 22140 raft_consensus.cc:2258] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P 723b2b9b57114b568c9fb840f98b38a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:19:05.896723 22140 ts_tablet_manager.cc:1905] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P 723b2b9b57114b568c9fb840f98b38a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:19:05.898409 22140 ts_tablet_manager.cc:1918] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P 723b2b9b57114b568c9fb840f98b38a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:19:05.898512 22140 log.cc:1196] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P 723b2b9b57114b568c9fb840f98b38a5: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/ts-1/wal/wals/2e29721cfa5f4dc7a63eb75b58bfadf2
I20260926 06:19:05.898756 22140 ts_tablet_manager.cc:1939] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P 723b2b9b57114b568c9fb840f98b38a5: Deleting consensus metadata
I20260926 06:19:05.899840 21626 catalog_manager.cc:4841] TS 723b2b9b57114b568c9fb840f98b38a5 (127.2.110.194:33787): tablet 2e29721cfa5f4dc7a63eb75b58bfadf2 (table test-table [id=fc8922bce7794ac3aa209852956a9b60]) successfully deleted
W20260926 06:19:05.901439 21624 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.110.193:39175 (error 108)
W20260926 06:19:05.902253 22110 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:19:05.901870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.193:39175 (local address 127.0.0.1:36322)
0926 06:19:05.901934 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:05.901948 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:05.901994 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:05.902186 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.193:39175: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 06:19:05.902446 21624 catalog_manager.cc:4584] TS 59a4d18422bf4313accfebe3c72c6387 (127.2.110.193:39175): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2e29721cfa5f4dc7a63eb75b58bfadf2: Network error: Client connection negotiation failed: client connection to 127.2.110.193:39175: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 06:19:05.902485 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 21835
I20260926 06:19:05.903255 22138 raft_consensus.cc:2258] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P fc4fc74e57f64094acfcecf84b8d94b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:19:05.904567 22138 ts_tablet_manager.cc:1905] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P fc4fc74e57f64094acfcecf84b8d94b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:19:05.906697 22138 ts_tablet_manager.cc:1918] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P fc4fc74e57f64094acfcecf84b8d94b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:19:05.906786 22138 log.cc:1196] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P fc4fc74e57f64094acfcecf84b8d94b8: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403292369990-2491-0/minicluster-data/ts-2/wal/wals/2e29721cfa5f4dc7a63eb75b58bfadf2
I20260926 06:19:05.907064 22138 ts_tablet_manager.cc:1939] T 2e29721cfa5f4dc7a63eb75b58bfadf2 P fc4fc74e57f64094acfcecf84b8d94b8: Deleting consensus metadata
I20260926 06:19:05.907601 21625 catalog_manager.cc:4841] TS fc4fc74e57f64094acfcecf84b8d94b8 (127.2.110.195:45783): tablet 2e29721cfa5f4dc7a63eb75b58bfadf2 (table test-table [id=fc8922bce7794ac3aa209852956a9b60]) successfully deleted
I20260926 06:19:05.908644 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 21969
I20260926 06:19:05.913610 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 21607
2026-09-26T06:19:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2044 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T06:19:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:19:05Z Disabled control of system clock
I20260926 06:19:05.943393 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:33615
--webserver_interface=127.2.110.254
--webserver_port=0
--builtin_ntp_servers=127.2.110.212:40183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.110.254:33615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 06:19:06.030678 22145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:19:06.030936 22145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:19:06.030959 22145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:19:06.032382 22145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:19:06.032423 22145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:19:06.032435 22145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:19:06.032449 22145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:19:06.032461 22145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:19:06.033978 22145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:40183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.110.254:33615
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:33615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:19:06.034394 22145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:19:06.034693 22145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:19:06.039863 22151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:06.039885 22145 server_base.cc:1034] running on GCE node
W20260926 06:19:06.039880 22150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:06.039863 22153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:06.040812 22145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:19:06.041154 22145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:19:06.042335 22145 hybrid_clock.cc:648] HybridClock initialized: now 1790403546042308 us; error 41 us; skew 500 ppm
I20260926 06:19:06.043931 22145 webserver.cc:460] Webserver started at http://127.2.110.254:43349/ using document root <none> and password file <none>
I20260926 06:19:06.044261 22145 fs_manager.cc:359] Metadata directory not provided
I20260926 06:19:06.044318 22145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:19:06.044476 22145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:19:06.045830 22145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/master-0/data/instance:
uuid: "5225b7f40e0d49c8a3f7cef3ac47d8e9"
format_stamp: "Formatted at 2026-09-26 06:19:06 on dist-test-slave-hndd"
server_key: "e8e1280faeac7b371629fd4506f5e05e"
server_key_iv: "94630419b7bb3d8b1de5175aa92d204f"
server_key_version: "encryptionkey@0"
I20260926 06:19:06.046344 22145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/master-0/wal/instance:
uuid: "5225b7f40e0d49c8a3f7cef3ac47d8e9"
format_stamp: "Formatted at 2026-09-26 06:19:06 on dist-test-slave-hndd"
server_key: "e8e1280faeac7b371629fd4506f5e05e"
server_key_iv: "94630419b7bb3d8b1de5175aa92d204f"
server_key_version: "encryptionkey@0"
I20260926 06:19:06.047959 22145 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:19:06.049445 22159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:06.049774 22145 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:19:06.049921 22145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "5225b7f40e0d49c8a3f7cef3ac47d8e9"
format_stamp: "Formatted at 2026-09-26 06:19:06 on dist-test-slave-hndd"
server_key: "e8e1280faeac7b371629fd4506f5e05e"
server_key_iv: "94630419b7bb3d8b1de5175aa92d204f"
server_key_version: "encryptionkey@0"
I20260926 06:19:06.050148 22145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:19:06.064651 22145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:19:06.065032 22145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:19:06.065193 22145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:19:06.069661 22145 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.254:33615
I20260926 06:19:06.069720 22211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.254:33615 every 8 connection(s)
I20260926 06:19:06.070091 22145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:19:06.071055 22212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:19:06.073454 22212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5225b7f40e0d49c8a3f7cef3ac47d8e9: Bootstrap starting.
I20260926 06:19:06.074069 22212 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5225b7f40e0d49c8a3f7cef3ac47d8e9: Neither blocks nor log segments found. Creating new log.
I20260926 06:19:06.074383 22212 log.cc:824] T 00000000000000000000000000000000 P 5225b7f40e0d49c8a3f7cef3ac47d8e9: Log is configured to *not* fsync() on all Append() calls
I20260926 06:19:06.075150 22212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5225b7f40e0d49c8a3f7cef3ac47d8e9: No bootstrap required, opened a new log
I20260926 06:19:06.076772 22212 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5225b7f40e0d49c8a3f7cef3ac47d8e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5225b7f40e0d49c8a3f7cef3ac47d8e9" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 33615 } }
I20260926 06:19:06.076926 22212 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5225b7f40e0d49c8a3f7cef3ac47d8e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:19:06.076949 22212 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5225b7f40e0d49c8a3f7cef3ac47d8e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5225b7f40e0d49c8a3f7cef3ac47d8e9, State: Initialized, Role: FOLLOWER
I20260926 06:19:06.077102 22212 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5225b7f40e0d49c8a3f7cef3ac47d8e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5225b7f40e0d49c8a3f7cef3ac47d8e9" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 33615 } }
I20260926 06:19:06.077178 22212 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5225b7f40e0d49c8a3f7cef3ac47d8e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:19:06.077200 22212 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5225b7f40e0d49c8a3f7cef3ac47d8e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:19:06.077229 22212 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5225b7f40e0d49c8a3f7cef3ac47d8e9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:19:06.077816 22212 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5225b7f40e0d49c8a3f7cef3ac47d8e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5225b7f40e0d49c8a3f7cef3ac47d8e9" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 33615 } }
I20260926 06:19:06.077903 22212 leader_election.cc:302] T 00000000000000000000000000000000 P 5225b7f40e0d49c8a3f7cef3ac47d8e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5225b7f40e0d49c8a3f7cef3ac47d8e9; no voters:
I20260926 06:19:06.078095 22212 leader_election.cc:288] T 00000000000000000000000000000000 P 5225b7f40e0d49c8a3f7cef3ac47d8e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:19:06.078189 22215 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5225b7f40e0d49c8a3f7cef3ac47d8e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:19:06.078382 22212 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5225b7f40e0d49c8a3f7cef3ac47d8e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:19:06.078389 22215 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5225b7f40e0d49c8a3f7cef3ac47d8e9 [term 1 LEADER]: Becoming Leader. State: Replica: 5225b7f40e0d49c8a3f7cef3ac47d8e9, State: Running, Role: LEADER
I20260926 06:19:06.078550 22215 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5225b7f40e0d49c8a3f7cef3ac47d8e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5225b7f40e0d49c8a3f7cef3ac47d8e9" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 33615 } }
I20260926 06:19:06.078977 22217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5225b7f40e0d49c8a3f7cef3ac47d8e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5225b7f40e0d49c8a3f7cef3ac47d8e9. Latest consensus state: current_term: 1 leader_uuid: "5225b7f40e0d49c8a3f7cef3ac47d8e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5225b7f40e0d49c8a3f7cef3ac47d8e9" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 33615 } } }
I20260926 06:19:06.079069 22216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5225b7f40e0d49c8a3f7cef3ac47d8e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5225b7f40e0d49c8a3f7cef3ac47d8e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5225b7f40e0d49c8a3f7cef3ac47d8e9" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 33615 } } }
I20260926 06:19:06.079087 22217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5225b7f40e0d49c8a3f7cef3ac47d8e9 [sys.catalog]: This master's current role is: LEADER
I20260926 06:19:06.079169 22216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5225b7f40e0d49c8a3f7cef3ac47d8e9 [sys.catalog]: This master's current role is: LEADER
I20260926 06:19:06.079471 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 22145
I20260926 06:19:06.079589 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/master-0/wal/instance
I20260926 06:19:06.079843 2491 external_mini_cluster.cc:1428] Setting key c2cb02258486511d3c03d76f2cdfca74
I20260926 06:19:06.079926 22224 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:19:06.080531 22224 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:19:06.082165 22224 catalog_manager.cc:1347] Generated new cluster ID: 78be2a8fdc7642d7b1d084ec65a1ce67
I20260926 06:19:06.082221 22224 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:19:06.086958 22234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:06.080397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39816 (local address 127.2.110.254:33615)
0926 06:19:06.080606 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:06.080615 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:06.080649 (+ 34us) server_negotiation.cc:407] Connection header received
0926 06:19:06.080769 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:06.080776 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:06.080867 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:06.081028 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:06.081300 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:06.082812 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:06.083998 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:06.085104 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:06.085265 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:06.086073 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:06.086077 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:06.086077 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:19:06.086319 (+ 242us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:06.086362 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:19:06.086378 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:06.086454 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:06.086520 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 06:19:06.086719 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":76,"threads_started":1}
I20260926 06:19:06.089517 22224 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:19:06.090054 22224 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:19:06.095677 22224 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5225b7f40e0d49c8a3f7cef3ac47d8e9: Generated new TSK 0
I20260926 06:19:06.095994 22224 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:19:06.099146 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.193:0
--local_ip_for_outbound_sockets=127.2.110.193
--webserver_interface=127.2.110.193
--webserver_port=0
--tserver_master_addrs=127.2.110.254:33615
--builtin_ntp_servers=127.2.110.212:40183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 06:19:06.189556 22236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:19:06.189774 22236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:19:06.189806 22236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:19:06.191069 22236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:19:06.191118 22236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:19:06.191145 22236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.193
I20260926 06:19:06.192582 22236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:40183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.110.193
--webserver_port=0
--tserver_master_addrs=127.2.110.254:33615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.193
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:19:06.192867 22236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:19:06.193204 22236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:19:06.198361 22242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:06.198436 22241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:06.198510 22236 server_base.cc:1034] running on GCE node
W20260926 06:19:06.198364 22244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:06.198783 22236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:19:06.199039 22236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:19:06.200232 22236 hybrid_clock.cc:648] HybridClock initialized: now 1790403546200215 us; error 30 us; skew 500 ppm
I20260926 06:19:06.201746 22236 webserver.cc:460] Webserver started at http://127.2.110.193:42985/ using document root <none> and password file <none>
I20260926 06:19:06.202044 22236 fs_manager.cc:359] Metadata directory not provided
I20260926 06:19:06.202099 22236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:19:06.202242 22236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:19:06.203434 22236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/ts-0/data/instance:
uuid: "101918afe7dd499ea8f3b8c25a305d35"
format_stamp: "Formatted at 2026-09-26 06:19:06 on dist-test-slave-hndd"
server_key: "c1a2b1e43bd50db3082dacf761cc695c"
server_key_iv: "b9b72cee0966364ab8d5ca1956bd16fd"
server_key_version: "encryptionkey@0"
I20260926 06:19:06.203814 22236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/ts-0/wal/instance:
uuid: "101918afe7dd499ea8f3b8c25a305d35"
format_stamp: "Formatted at 2026-09-26 06:19:06 on dist-test-slave-hndd"
server_key: "c1a2b1e43bd50db3082dacf761cc695c"
server_key_iv: "b9b72cee0966364ab8d5ca1956bd16fd"
server_key_version: "encryptionkey@0"
I20260926 06:19:06.205861 22236 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260926 06:19:06.206821 22250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:06.207062 22236 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 06:19:06.207267 22236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/ts-0/wal
uuid: "101918afe7dd499ea8f3b8c25a305d35"
format_stamp: "Formatted at 2026-09-26 06:19:06 on dist-test-slave-hndd"
server_key: "c1a2b1e43bd50db3082dacf761cc695c"
server_key_iv: "b9b72cee0966364ab8d5ca1956bd16fd"
server_key_version: "encryptionkey@0"
I20260926 06:19:06.207378 22236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:19:06.226748 22236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:19:06.227151 22236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:19:06.227320 22236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:19:06.227578 22236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:19:06.227921 22236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:19:06.227981 22236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:06.228027 22236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:19:06.228051 22236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:06.234138 22236 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.193:46171
I20260926 06:19:06.234189 22363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.193:46171 every 8 connection(s)
I20260926 06:19:06.234541 22236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/ts-0/data/info.pb
I20260926 06:19:06.235194 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 22236
I20260926 06:19:06.235321 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/ts-0/wal/instance
I20260926 06:19:06.235534 2491 external_mini_cluster.cc:1428] Setting key eb889bce11ff2799220786dd4be64376
I20260926 06:19:06.236557 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.194:0
--local_ip_for_outbound_sockets=127.2.110.194
--webserver_interface=127.2.110.194
--webserver_port=0
--tserver_master_addrs=127.2.110.254:33615
--builtin_ntp_servers=127.2.110.212:40183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 06:19:06.240444 22234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:06.235384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.193:44503 (local address 127.2.110.254:33615)
0926 06:19:06.235470 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:06.235473 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:06.235733 (+ 260us) server_negotiation.cc:407] Connection header received
0926 06:19:06.236267 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:06.236268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:06.236281 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:06.236332 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:06.237105 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:06.237723 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:06.238881 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:06.239105 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:06.239188 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:06.240128 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:06.240129 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:06.240129 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:19:06.240134 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:06.240150 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:19:06.240155 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:06.240226 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:06.240321 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 06:19:06.240358 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 06:19:06.240680 22366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:06.235327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:33615 (local address 127.2.110.193:44503)
0926 06:19:06.235565 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:06.235620 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:06.236156 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:06.236401 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:19:06.236409 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:19:06.236418 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:19:06.237015 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:06.237019 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:06.237825 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:06.237826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:06.238749 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:06.238752 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:06.239209 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:06.239210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:06.239346 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:06.239976 (+ 630us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:19:06.239977 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:19:06.240006 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:19:06.240031 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:19:06.240228 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:19:06.240235 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:19:06.240239 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:19:06.240325 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 06:19:06.240438 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":116,"threads_started":1}
I20260926 06:19:06.241240 22364 heartbeater.cc:347] Connected to a master server at 127.2.110.254:33615
I20260926 06:19:06.241391 22364 heartbeater.cc:464] Registering TS with master...
I20260926 06:19:06.241629 22364 heartbeater.cc:511] Master 127.2.110.254:33615 requested a full tablet report, sending...
I20260926 06:19:06.242156 22176 ts_manager.cc:194] Registered new tserver with Master: 101918afe7dd499ea8f3b8c25a305d35 (127.2.110.193:46171)
I20260926 06:19:06.242796 22176 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.110.193:44503
W20260926 06:19:06.320498 22367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:19:06.320734 22367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:19:06.320758 22367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:19:06.322028 22367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:19:06.322078 22367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:19:06.322106 22367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.194
I20260926 06:19:06.323580 22367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:40183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.110.194
--webserver_port=0
--tserver_master_addrs=127.2.110.254:33615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.194
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:19:06.323865 22367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:19:06.324286 22367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:19:06.329766 22373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:06.329754 22372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:06.329754 22367 server_base.cc:1034] running on GCE node
W20260926 06:19:06.329789 22375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:06.330286 22367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:19:06.330538 22367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:19:06.331710 22367 hybrid_clock.cc:648] HybridClock initialized: now 1790403546331698 us; error 29 us; skew 500 ppm
I20260926 06:19:06.333173 22367 webserver.cc:460] Webserver started at http://127.2.110.194:32933/ using document root <none> and password file <none>
I20260926 06:19:06.333534 22367 fs_manager.cc:359] Metadata directory not provided
I20260926 06:19:06.333611 22367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:19:06.333755 22367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:19:06.334877 22367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/ts-1/data/instance:
uuid: "fce62f88fa9b4f368a39e307e8316cb4"
format_stamp: "Formatted at 2026-09-26 06:19:06 on dist-test-slave-hndd"
server_key: "8fbcec3e69c79e7385e4701a8330708b"
server_key_iv: "5a81a59610d5508a072ea38f7314341e"
server_key_version: "encryptionkey@0"
I20260926 06:19:06.335253 22367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/ts-1/wal/instance:
uuid: "fce62f88fa9b4f368a39e307e8316cb4"
format_stamp: "Formatted at 2026-09-26 06:19:06 on dist-test-slave-hndd"
server_key: "8fbcec3e69c79e7385e4701a8330708b"
server_key_iv: "5a81a59610d5508a072ea38f7314341e"
server_key_version: "encryptionkey@0"
I20260926 06:19:06.336644 22367 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:19:06.337591 22381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:06.337801 22367 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:19:06.337913 22367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/ts-1/wal
uuid: "fce62f88fa9b4f368a39e307e8316cb4"
format_stamp: "Formatted at 2026-09-26 06:19:06 on dist-test-slave-hndd"
server_key: "8fbcec3e69c79e7385e4701a8330708b"
server_key_iv: "5a81a59610d5508a072ea38f7314341e"
server_key_version: "encryptionkey@0"
I20260926 06:19:06.338004 22367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:19:06.350741 22367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:19:06.351118 22367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:19:06.351289 22367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:19:06.351550 22367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:19:06.351900 22367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:19:06.351965 22367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:06.352011 22367 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:19:06.352036 22367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:06.358325 22367 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.194:35315
I20260926 06:19:06.358376 22494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.194:35315 every 8 connection(s)
I20260926 06:19:06.358714 22367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/ts-1/data/info.pb
I20260926 06:19:06.361733 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 22367
I20260926 06:19:06.361867 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/ts-1/wal/instance
I20260926 06:19:06.362082 2491 external_mini_cluster.cc:1428] Setting key a596c61443edb459afce5a30a91a5aa1
I20260926 06:19:06.363345 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.110.195:0
--local_ip_for_outbound_sockets=127.2.110.195
--webserver_interface=127.2.110.195
--webserver_port=0
--tserver_master_addrs=127.2.110.254:33615
--builtin_ntp_servers=127.2.110.212:40183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 06:19:06.364718 22234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:06.359498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.194:46779 (local address 127.2.110.254:33615)
0926 06:19:06.359575 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:06.359578 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:06.359853 (+ 275us) server_negotiation.cc:407] Connection header received
0926 06:19:06.360370 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:06.360371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:06.360381 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:06.360437 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:06.361126 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:06.361738 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:06.363096 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:06.363321 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:06.363398 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:06.364374 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:06.364376 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:06.364376 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:19:06.364380 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:06.364395 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:19:06.364399 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:06.364447 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:06.364561 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 06:19:06.364586 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 06:19:06.364938 22497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:06.359468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:33615 (local address 127.2.110.194:46779)
0926 06:19:06.359702 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:06.359737 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:06.360260 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:06.360512 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:19:06.360516 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:19:06.360523 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:19:06.361053 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:06.361057 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:06.361855 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:06.361857 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:06.362959 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:06.362963 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:06.363429 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:06.363430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:06.363598 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:06.364199 (+ 601us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:19:06.364200 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:19:06.364239 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:19:06.364269 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:19:06.364465 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:19:06.364471 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:19:06.364474 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:19:06.364556 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 06:19:06.364675 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":99,"threads_started":1}
I20260926 06:19:06.365428 22495 heartbeater.cc:347] Connected to a master server at 127.2.110.254:33615
I20260926 06:19:06.365540 22495 heartbeater.cc:464] Registering TS with master...
I20260926 06:19:06.365742 22495 heartbeater.cc:511] Master 127.2.110.254:33615 requested a full tablet report, sending...
I20260926 06:19:06.366168 22176 ts_manager.cc:194] Registered new tserver with Master: fce62f88fa9b4f368a39e307e8316cb4 (127.2.110.194:35315)
I20260926 06:19:06.366706 22176 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.110.194:46779
W20260926 06:19:06.449252 22498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:19:06.449544 22498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:19:06.449568 22498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:19:06.450714 22498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:19:06.450754 22498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:19:06.450780 22498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.110.195
I20260926 06:19:06.452080 22498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:40183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.110.195
--webserver_port=0
--tserver_master_addrs=127.2.110.254:33615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.110.195
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:19:06.452301 22498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:19:06.452550 22498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:19:06.457873 22503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:06.457899 22506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:06.457912 22504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:06.457931 22498 server_base.cc:1034] running on GCE node
I20260926 06:19:06.458180 22498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:19:06.458451 22498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:19:06.459616 22498 hybrid_clock.cc:648] HybridClock initialized: now 1790403546459594 us; error 37 us; skew 500 ppm
I20260926 06:19:06.460963 22498 webserver.cc:460] Webserver started at http://127.2.110.195:41785/ using document root <none> and password file <none>
I20260926 06:19:06.461239 22498 fs_manager.cc:359] Metadata directory not provided
I20260926 06:19:06.461341 22498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:19:06.461539 22498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:19:06.462661 22498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/ts-2/data/instance:
uuid: "176dd1551f3e4f3baa6907a309ccebbb"
format_stamp: "Formatted at 2026-09-26 06:19:06 on dist-test-slave-hndd"
server_key: "b984bb2e65ca51483776d5af8933ed84"
server_key_iv: "102986a0fc5e7bba3c5a395020ea4784"
server_key_version: "encryptionkey@0"
I20260926 06:19:06.463043 22498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/ts-2/wal/instance:
uuid: "176dd1551f3e4f3baa6907a309ccebbb"
format_stamp: "Formatted at 2026-09-26 06:19:06 on dist-test-slave-hndd"
server_key: "b984bb2e65ca51483776d5af8933ed84"
server_key_iv: "102986a0fc5e7bba3c5a395020ea4784"
server_key_version: "encryptionkey@0"
I20260926 06:19:06.464461 22498 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:19:06.465481 22512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:06.465806 22498 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:19:06.465920 22498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/ts-2/wal
uuid: "176dd1551f3e4f3baa6907a309ccebbb"
format_stamp: "Formatted at 2026-09-26 06:19:06 on dist-test-slave-hndd"
server_key: "b984bb2e65ca51483776d5af8933ed84"
server_key_iv: "102986a0fc5e7bba3c5a395020ea4784"
server_key_version: "encryptionkey@0"
I20260926 06:19:06.466017 22498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:19:06.493543 22498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:19:06.493899 22498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:19:06.494062 22498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:19:06.494351 22498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:19:06.494705 22498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:19:06.494763 22498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:06.494800 22498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:19:06.494829 22498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:06.501051 22498 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.195:34345
I20260926 06:19:06.501096 22625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.195:34345 every 8 connection(s)
I20260926 06:19:06.501511 22498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/ts-2/data/info.pb
I20260926 06:19:06.507526 22234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:06.502450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:46095 (local address 127.2.110.254:33615)
0926 06:19:06.502505 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:06.502508 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:06.502717 (+ 209us) server_negotiation.cc:407] Connection header received
0926 06:19:06.503357 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:06.503359 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:06.503376 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:06.503479 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:06.504239 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:06.504808 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:06.505921 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:06.506129 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:06.506210 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:06.507200 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:06.507201 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:06.507202 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:19:06.507207 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:06.507223 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:19:06.507255 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:06.507312 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:06.507424 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 06:19:06.507455 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 06:19:06.507748 22628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:06.502372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:33615 (local address 127.2.110.195:46095)
0926 06:19:06.502571 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:06.502612 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:06.503115 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:06.503562 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:19:06.503565 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:19:06.503573 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:19:06.504090 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:06.504094 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:06.504913 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:06.504914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:06.505822 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:06.505825 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:06.506255 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:06.506256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:06.506395 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:06.507024 (+ 629us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:19:06.507024 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:19:06.507054 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:19:06.507080 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:19:06.507327 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:19:06.507332 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:19:06.507335 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:19:06.507405 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 06:19:06.507494 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":97,"threads_started":1}
I20260926 06:19:06.508308 22626 heartbeater.cc:347] Connected to a master server at 127.2.110.254:33615
I20260926 06:19:06.508427 22626 heartbeater.cc:464] Registering TS with master...
I20260926 06:19:06.508632 22626 heartbeater.cc:511] Master 127.2.110.254:33615 requested a full tablet report, sending...
I20260926 06:19:06.509052 22176 ts_manager.cc:194] Registered new tserver with Master: 176dd1551f3e4f3baa6907a309ccebbb (127.2.110.195:34345)
I20260926 06:19:06.509569 22176 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.110.195:46095
I20260926 06:19:06.509866 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 22498
I20260926 06:19:06.509979 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/ts-2/wal/instance
I20260926 06:19:06.510191 2491 external_mini_cluster.cc:1428] Setting key 93ae91044fe07b621d5cff85a319c7ae
I20260926 06:19:06.511148 2491 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 06:19:06.515842 22234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:06.512677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53394 (local address 127.2.110.254:33615)
0926 06:19:06.512811 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:06.512813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:06.512824 (+ 11us) server_negotiation.cc:407] Connection header received
0926 06:19:06.512920 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:06.512921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:06.512934 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:06.512981 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:06.513113 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:06.513713 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:06.514555 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:06.514764 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:06.514825 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:06.515632 (+ 807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:06.515633 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:06.515633 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:19:06.515638 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:06.515651 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:19:06.515655 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:06.515713 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:06.515760 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 06:19:06.515784 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260926 06:19:06.517604 22176 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53394:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:19:06.519332 22176 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:19:06.526413 22634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:06.521649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.193:46171 (local address 127.0.0.1:44558)
0926 06:19:06.522450 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:06.522457 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:06.522533 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:06.522777 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:19:06.522778 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:19:06.522780 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:19:06.522830 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:06.522831 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:06.523929 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:06.523930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:06.525012 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:06.525015 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:06.526075 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:06.526076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:06.526152 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:06.526167 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 06:19:06.526246 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 06:19:06.526303 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"mutex_wait_us":5,"thread_start_us":97,"threads_started":1}
I20260926 06:19:06.526428 22638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:06.521898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44558 (local address 127.2.110.193:46171)
0926 06:19:06.522080 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:06.522085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:06.522505 (+ 420us) server_negotiation.cc:407] Connection header received
0926 06:19:06.522624 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:06.522627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:06.522642 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:06.522695 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:19:06.522909 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:06.523814 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:06.525110 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:06.525870 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:06.525940 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:06.526038 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:06.526281 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 06:19:06.526327 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":113,"threads_started":1}
I20260926 06:19:06.526679 22636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:06.521638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.195:34345 (local address 127.0.0.1:42250)
0926 06:19:06.522326 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:06.522335 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:06.522372 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:06.523152 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:19:06.523154 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:19:06.523156 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:19:06.523196 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:06.523197 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:06.524801 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:06.524801 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:06.525665 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:06.525667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:06.526502 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:06.526503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:06.526570 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:06.526575 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 06:19:06.526601 (+ 26us) client_negotiation.cc:240] Negotiation successful
0926 06:19:06.526632 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":73,"threads_started":1}
I20260926 06:19:06.527082 22637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:06.521729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42250 (local address 127.2.110.195:34345)
0926 06:19:06.522963 (+ 1234us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:06.522984 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:06.522995 (+ 11us) server_negotiation.cc:407] Connection header received
0926 06:19:06.523037 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:06.523039 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:06.523053 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:06.523128 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:19:06.523281 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:06.524730 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:06.525752 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:06.526395 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:06.526808 (+ 413us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:06.526922 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:06.526950 (+ 28us) server_negotiation.cc:299] Negotiation successful
0926 06:19:06.526999 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1196,"thread_start_us":88,"threads_started":1}
I20260926 06:19:06.527238 22298 tablet_service.cc:1467] Processing CreateTablet for tablet 8ae9fb939ea14025a77b5e34516dcf78 (DEFAULT_TABLE table=test-table [id=944a1d61d13344718607795c88618fb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:19:06.527123 22639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:06.521980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50984 (local address 127.2.110.194:35315)
0926 06:19:06.522287 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:06.522292 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:06.522331 (+ 39us) server_negotiation.cc:407] Connection header received
0926 06:19:06.522374 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:06.522376 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:06.522394 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:06.522473 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:19:06.522711 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:06.524054 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:06.525437 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:06.526171 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:06.526455 (+ 284us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:06.526978 (+ 523us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:06.527005 (+ 27us) server_negotiation.cc:299] Negotiation successful
0926 06:19:06.527047 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":200,"threads_started":1}
I20260926 06:19:06.527662 22298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ae9fb939ea14025a77b5e34516dcf78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:19:06.527784 22560 tablet_service.cc:1467] Processing CreateTablet for tablet 8ae9fb939ea14025a77b5e34516dcf78 (DEFAULT_TABLE table=test-table [id=944a1d61d13344718607795c88618fb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:19:06.528117 22560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ae9fb939ea14025a77b5e34516dcf78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:19:06.530391 22640 tablet_bootstrap.cc:492] T 8ae9fb939ea14025a77b5e34516dcf78 P 101918afe7dd499ea8f3b8c25a305d35: Bootstrap starting.
I20260926 06:19:06.530584 22635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:06.521649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.194:35315 (local address 127.0.0.1:50984)
0926 06:19:06.522111 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:06.522149 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:06.522245 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:06.522551 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:19:06.522554 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:19:06.522556 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:19:06.522628 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:06.522630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:06.524315 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:06.524316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:06.525310 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:06.525313 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:06.526267 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:06.526268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:06.526319 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:06.526322 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:19:06.526351 (+ 29us) client_negotiation.cc:240] Negotiation successful
0926 06:19:06.526381 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":104,"threads_started":1}
I20260926 06:19:06.530942 22641 tablet_bootstrap.cc:492] T 8ae9fb939ea14025a77b5e34516dcf78 P 176dd1551f3e4f3baa6907a309ccebbb: Bootstrap starting.
I20260926 06:19:06.531244 22640 tablet_bootstrap.cc:654] T 8ae9fb939ea14025a77b5e34516dcf78 P 101918afe7dd499ea8f3b8c25a305d35: Neither blocks nor log segments found. Creating new log.
I20260926 06:19:06.531343 22429 tablet_service.cc:1467] Processing CreateTablet for tablet 8ae9fb939ea14025a77b5e34516dcf78 (DEFAULT_TABLE table=test-table [id=944a1d61d13344718607795c88618fb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:19:06.531581 22640 log.cc:824] T 8ae9fb939ea14025a77b5e34516dcf78 P 101918afe7dd499ea8f3b8c25a305d35: Log is configured to *not* fsync() on all Append() calls
I20260926 06:19:06.531658 22641 tablet_bootstrap.cc:654] T 8ae9fb939ea14025a77b5e34516dcf78 P 176dd1551f3e4f3baa6907a309ccebbb: Neither blocks nor log segments found. Creating new log.
I20260926 06:19:06.531706 22429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ae9fb939ea14025a77b5e34516dcf78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:19:06.531947 22641 log.cc:824] T 8ae9fb939ea14025a77b5e34516dcf78 P 176dd1551f3e4f3baa6907a309ccebbb: Log is configured to *not* fsync() on all Append() calls
I20260926 06:19:06.532636 22641 tablet_bootstrap.cc:492] T 8ae9fb939ea14025a77b5e34516dcf78 P 176dd1551f3e4f3baa6907a309ccebbb: No bootstrap required, opened a new log
I20260926 06:19:06.532724 22641 ts_tablet_manager.cc:1397] T 8ae9fb939ea14025a77b5e34516dcf78 P 176dd1551f3e4f3baa6907a309ccebbb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 06:19:06.533975 22640 tablet_bootstrap.cc:492] T 8ae9fb939ea14025a77b5e34516dcf78 P 101918afe7dd499ea8f3b8c25a305d35: No bootstrap required, opened a new log
I20260926 06:19:06.534046 22640 ts_tablet_manager.cc:1397] T 8ae9fb939ea14025a77b5e34516dcf78 P 101918afe7dd499ea8f3b8c25a305d35: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260926 06:19:06.535032 22644 tablet_bootstrap.cc:492] T 8ae9fb939ea14025a77b5e34516dcf78 P fce62f88fa9b4f368a39e307e8316cb4: Bootstrap starting.
I20260926 06:19:06.535303 22641 raft_consensus.cc:348] T 8ae9fb939ea14025a77b5e34516dcf78 P 176dd1551f3e4f3baa6907a309ccebbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "101918afe7dd499ea8f3b8c25a305d35" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46171 } } peers { permanent_uuid: "176dd1551f3e4f3baa6907a309ccebbb" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34345 } } peers { permanent_uuid: "fce62f88fa9b4f368a39e307e8316cb4" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 35315 } }
I20260926 06:19:06.535501 22641 raft_consensus.cc:374] T 8ae9fb939ea14025a77b5e34516dcf78 P 176dd1551f3e4f3baa6907a309ccebbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:19:06.535550 22641 raft_consensus.cc:729] T 8ae9fb939ea14025a77b5e34516dcf78 P 176dd1551f3e4f3baa6907a309ccebbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 176dd1551f3e4f3baa6907a309ccebbb, State: Initialized, Role: FOLLOWER
I20260926 06:19:06.535686 22641 consensus_queue.cc:260] T 8ae9fb939ea14025a77b5e34516dcf78 P 176dd1551f3e4f3baa6907a309ccebbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "101918afe7dd499ea8f3b8c25a305d35" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46171 } } peers { permanent_uuid: "176dd1551f3e4f3baa6907a309ccebbb" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34345 } } peers { permanent_uuid: "fce62f88fa9b4f368a39e307e8316cb4" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 35315 } }
I20260926 06:19:06.535951 22644 tablet_bootstrap.cc:654] T 8ae9fb939ea14025a77b5e34516dcf78 P fce62f88fa9b4f368a39e307e8316cb4: Neither blocks nor log segments found. Creating new log.
I20260926 06:19:06.535971 22641 ts_tablet_manager.cc:1428] T 8ae9fb939ea14025a77b5e34516dcf78 P 176dd1551f3e4f3baa6907a309ccebbb: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 06:19:06.536016 22626 heartbeater.cc:503] Master 127.2.110.254:33615 was elected leader, sending a full tablet report...
I20260926 06:19:06.536278 22644 log.cc:824] T 8ae9fb939ea14025a77b5e34516dcf78 P fce62f88fa9b4f368a39e307e8316cb4: Log is configured to *not* fsync() on all Append() calls
I20260926 06:19:06.536685 22640 raft_consensus.cc:348] T 8ae9fb939ea14025a77b5e34516dcf78 P 101918afe7dd499ea8f3b8c25a305d35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "101918afe7dd499ea8f3b8c25a305d35" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46171 } } peers { permanent_uuid: "176dd1551f3e4f3baa6907a309ccebbb" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34345 } } peers { permanent_uuid: "fce62f88fa9b4f368a39e307e8316cb4" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 35315 } }
I20260926 06:19:06.536871 22640 raft_consensus.cc:374] T 8ae9fb939ea14025a77b5e34516dcf78 P 101918afe7dd499ea8f3b8c25a305d35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:19:06.536922 22640 raft_consensus.cc:729] T 8ae9fb939ea14025a77b5e34516dcf78 P 101918afe7dd499ea8f3b8c25a305d35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 101918afe7dd499ea8f3b8c25a305d35, State: Initialized, Role: FOLLOWER
I20260926 06:19:06.537036 22640 consensus_queue.cc:260] T 8ae9fb939ea14025a77b5e34516dcf78 P 101918afe7dd499ea8f3b8c25a305d35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "101918afe7dd499ea8f3b8c25a305d35" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46171 } } peers { permanent_uuid: "176dd1551f3e4f3baa6907a309ccebbb" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34345 } } peers { permanent_uuid: "fce62f88fa9b4f368a39e307e8316cb4" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 35315 } }
I20260926 06:19:06.537081 22644 tablet_bootstrap.cc:492] T 8ae9fb939ea14025a77b5e34516dcf78 P fce62f88fa9b4f368a39e307e8316cb4: No bootstrap required, opened a new log
I20260926 06:19:06.537175 22644 ts_tablet_manager.cc:1397] T 8ae9fb939ea14025a77b5e34516dcf78 P fce62f88fa9b4f368a39e307e8316cb4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 06:19:06.537322 22640 ts_tablet_manager.cc:1428] T 8ae9fb939ea14025a77b5e34516dcf78 P 101918afe7dd499ea8f3b8c25a305d35: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260926 06:19:06.537479 22364 heartbeater.cc:503] Master 127.2.110.254:33615 was elected leader, sending a full tablet report...
I20260926 06:19:06.539345 22644 raft_consensus.cc:348] T 8ae9fb939ea14025a77b5e34516dcf78 P fce62f88fa9b4f368a39e307e8316cb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "101918afe7dd499ea8f3b8c25a305d35" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46171 } } peers { permanent_uuid: "176dd1551f3e4f3baa6907a309ccebbb" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34345 } } peers { permanent_uuid: "fce62f88fa9b4f368a39e307e8316cb4" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 35315 } }
I20260926 06:19:06.539486 22644 raft_consensus.cc:374] T 8ae9fb939ea14025a77b5e34516dcf78 P fce62f88fa9b4f368a39e307e8316cb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:19:06.539508 22644 raft_consensus.cc:729] T 8ae9fb939ea14025a77b5e34516dcf78 P fce62f88fa9b4f368a39e307e8316cb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fce62f88fa9b4f368a39e307e8316cb4, State: Initialized, Role: FOLLOWER
I20260926 06:19:06.539634 22644 consensus_queue.cc:260] T 8ae9fb939ea14025a77b5e34516dcf78 P fce62f88fa9b4f368a39e307e8316cb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "101918afe7dd499ea8f3b8c25a305d35" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46171 } } peers { permanent_uuid: "176dd1551f3e4f3baa6907a309ccebbb" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34345 } } peers { permanent_uuid: "fce62f88fa9b4f368a39e307e8316cb4" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 35315 } }
I20260926 06:19:06.539927 22644 ts_tablet_manager.cc:1428] T 8ae9fb939ea14025a77b5e34516dcf78 P fce62f88fa9b4f368a39e307e8316cb4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 06:19:06.540001 22495 heartbeater.cc:503] Master 127.2.110.254:33615 was elected leader, sending a full tablet report...
W20260926 06:19:06.609409 22496 tablet.cc:2369] T 8ae9fb939ea14025a77b5e34516dcf78 P fce62f88fa9b4f368a39e307e8316cb4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:19:06.706420 22645 raft_consensus.cc:482] T 8ae9fb939ea14025a77b5e34516dcf78 P 176dd1551f3e4f3baa6907a309ccebbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:19:06.706594 22645 raft_consensus.cc:504] T 8ae9fb939ea14025a77b5e34516dcf78 P 176dd1551f3e4f3baa6907a309ccebbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "101918afe7dd499ea8f3b8c25a305d35" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46171 } } peers { permanent_uuid: "176dd1551f3e4f3baa6907a309ccebbb" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34345 } } peers { permanent_uuid: "fce62f88fa9b4f368a39e307e8316cb4" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 35315 } }
I20260926 06:19:06.707036 22645 leader_election.cc:288] T 8ae9fb939ea14025a77b5e34516dcf78 P 176dd1551f3e4f3baa6907a309ccebbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 101918afe7dd499ea8f3b8c25a305d35 (127.2.110.193:46171), fce62f88fa9b4f368a39e307e8316cb4 (127.2.110.194:35315)
I20260926 06:19:06.710559 22638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:06.707273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:48351 (local address 127.2.110.193:46171)
0926 06:19:06.707321 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:06.707323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:06.707335 (+ 12us) server_negotiation.cc:407] Connection header received
0926 06:19:06.707424 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:06.707426 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:06.707439 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:06.707496 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:19:06.707686 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:06.708233 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:06.709626 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:06.710172 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:06.710231 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:06.710239 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:06.710426 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 06:19:06.710468 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 06:19:06.710559 22628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:06.707177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.193:46171 (local address 127.2.110.195:48351)
0926 06:19:06.707253 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:06.707266 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:06.707329 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:06.707522 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:19:06.707524 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:19:06.707527 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:19:06.707605 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:06.707608 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:06.708370 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:06.708373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:06.709537 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:06.709541 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:06.710249 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:06.710250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:06.710325 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:06.710329 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:19:06.710395 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 06:19:06.710459 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 06:19:06.710872 22649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:06.707215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.194:35315 (local address 127.2.110.195:53803)
0926 06:19:06.707386 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:06.707403 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:06.707465 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:06.707633 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:19:06.707633 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:19:06.707635 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:19:06.707727 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:06.707728 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:06.708836 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:06.708838 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:06.709793 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:06.709796 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:06.710567 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:06.710569 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:06.710655 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:06.710660 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 06:19:06.710729 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 06:19:06.710775 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":111,"threads_started":1}
I20260926 06:19:06.710873 22639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:06.707304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.110.195:53803 (local address 127.2.110.194:35315)
0926 06:19:06.707380 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:06.707382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:06.707518 (+ 136us) server_negotiation.cc:407] Connection header received
0926 06:19:06.707545 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:06.707546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:06.707558 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:06.707617 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:19:06.707888 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:06.708712 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:06.709896 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:06.710483 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:06.710549 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:06.710557 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:06.710748 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 06:19:06.710806 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 06:19:06.710965 22318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ae9fb939ea14025a77b5e34516dcf78" candidate_uuid: "176dd1551f3e4f3baa6907a309ccebbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "101918afe7dd499ea8f3b8c25a305d35" is_pre_election: true
I20260926 06:19:06.711123 22318 raft_consensus.cc:2454] T 8ae9fb939ea14025a77b5e34516dcf78 P 101918afe7dd499ea8f3b8c25a305d35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 176dd1551f3e4f3baa6907a309ccebbb in term 0.
I20260926 06:19:06.711356 22513 leader_election.cc:302] T 8ae9fb939ea14025a77b5e34516dcf78 P 176dd1551f3e4f3baa6907a309ccebbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 101918afe7dd499ea8f3b8c25a305d35, 176dd1551f3e4f3baa6907a309ccebbb; no voters:
I20260926 06:19:06.711511 22645 raft_consensus.cc:2781] T 8ae9fb939ea14025a77b5e34516dcf78 P 176dd1551f3e4f3baa6907a309ccebbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:19:06.711580 22645 raft_consensus.cc:482] T 8ae9fb939ea14025a77b5e34516dcf78 P 176dd1551f3e4f3baa6907a309ccebbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:19:06.711606 22645 raft_consensus.cc:3037] T 8ae9fb939ea14025a77b5e34516dcf78 P 176dd1551f3e4f3baa6907a309ccebbb [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:19:06.711463 22449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ae9fb939ea14025a77b5e34516dcf78" candidate_uuid: "176dd1551f3e4f3baa6907a309ccebbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fce62f88fa9b4f368a39e307e8316cb4" is_pre_election: true
I20260926 06:19:06.711808 22449 raft_consensus.cc:2454] T 8ae9fb939ea14025a77b5e34516dcf78 P fce62f88fa9b4f368a39e307e8316cb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 176dd1551f3e4f3baa6907a309ccebbb in term 0.
I20260926 06:19:06.712570 22645 raft_consensus.cc:504] T 8ae9fb939ea14025a77b5e34516dcf78 P 176dd1551f3e4f3baa6907a309ccebbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "101918afe7dd499ea8f3b8c25a305d35" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46171 } } peers { permanent_uuid: "176dd1551f3e4f3baa6907a309ccebbb" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34345 } } peers { permanent_uuid: "fce62f88fa9b4f368a39e307e8316cb4" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 35315 } }
I20260926 06:19:06.712745 22645 leader_election.cc:288] T 8ae9fb939ea14025a77b5e34516dcf78 P 176dd1551f3e4f3baa6907a309ccebbb [CANDIDATE]: Term 1 election: Requested vote from peers 101918afe7dd499ea8f3b8c25a305d35 (127.2.110.193:46171), fce62f88fa9b4f368a39e307e8316cb4 (127.2.110.194:35315)
I20260926 06:19:06.712955 22449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ae9fb939ea14025a77b5e34516dcf78" candidate_uuid: "176dd1551f3e4f3baa6907a309ccebbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fce62f88fa9b4f368a39e307e8316cb4"
I20260926 06:19:06.712955 22318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ae9fb939ea14025a77b5e34516dcf78" candidate_uuid: "176dd1551f3e4f3baa6907a309ccebbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "101918afe7dd499ea8f3b8c25a305d35"
I20260926 06:19:06.713049 22318 raft_consensus.cc:3037] T 8ae9fb939ea14025a77b5e34516dcf78 P 101918afe7dd499ea8f3b8c25a305d35 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:19:06.713049 22449 raft_consensus.cc:3037] T 8ae9fb939ea14025a77b5e34516dcf78 P fce62f88fa9b4f368a39e307e8316cb4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:19:06.714016 22318 raft_consensus.cc:2454] T 8ae9fb939ea14025a77b5e34516dcf78 P 101918afe7dd499ea8f3b8c25a305d35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 176dd1551f3e4f3baa6907a309ccebbb in term 1.
I20260926 06:19:06.714020 22449 raft_consensus.cc:2454] T 8ae9fb939ea14025a77b5e34516dcf78 P fce62f88fa9b4f368a39e307e8316cb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 176dd1551f3e4f3baa6907a309ccebbb in term 1.
I20260926 06:19:06.714234 22515 leader_election.cc:302] T 8ae9fb939ea14025a77b5e34516dcf78 P 176dd1551f3e4f3baa6907a309ccebbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 176dd1551f3e4f3baa6907a309ccebbb, fce62f88fa9b4f368a39e307e8316cb4; no voters:
I20260926 06:19:06.714425 22645 raft_consensus.cc:2781] T 8ae9fb939ea14025a77b5e34516dcf78 P 176dd1551f3e4f3baa6907a309ccebbb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:19:06.714659 22645 raft_consensus.cc:686] T 8ae9fb939ea14025a77b5e34516dcf78 P 176dd1551f3e4f3baa6907a309ccebbb [term 1 LEADER]: Becoming Leader. State: Replica: 176dd1551f3e4f3baa6907a309ccebbb, State: Running, Role: LEADER
I20260926 06:19:06.714818 22645 consensus_queue.cc:237] T 8ae9fb939ea14025a77b5e34516dcf78 P 176dd1551f3e4f3baa6907a309ccebbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "101918afe7dd499ea8f3b8c25a305d35" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46171 } } peers { permanent_uuid: "176dd1551f3e4f3baa6907a309ccebbb" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34345 } } peers { permanent_uuid: "fce62f88fa9b4f368a39e307e8316cb4" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 35315 } }
I20260926 06:19:06.715924 22175 catalog_manager.cc:5495] T 8ae9fb939ea14025a77b5e34516dcf78 P 176dd1551f3e4f3baa6907a309ccebbb reported cstate change: term changed from 0 to 1, leader changed from <none> to 176dd1551f3e4f3baa6907a309ccebbb (127.2.110.195). New cstate: current_term: 1 leader_uuid: "176dd1551f3e4f3baa6907a309ccebbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "101918afe7dd499ea8f3b8c25a305d35" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "176dd1551f3e4f3baa6907a309ccebbb" member_type: VOTER last_known_addr { host: "127.2.110.195" port: 34345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fce62f88fa9b4f368a39e307e8316cb4" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 35315 } health_report { overall_health: UNKNOWN } } }
W20260926 06:19:06.735484 22365 tablet.cc:2369] T 8ae9fb939ea14025a77b5e34516dcf78 P 101918afe7dd499ea8f3b8c25a305d35: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 06:19:06.752344 22627 tablet.cc:2369] T 8ae9fb939ea14025a77b5e34516dcf78 P 176dd1551f3e4f3baa6907a309ccebbb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:19:06.754594 22637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:06.751986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42252 (local address 127.2.110.195:34345)
0926 06:19:06.752053 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:06.752055 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:06.752066 (+ 11us) server_negotiation.cc:407] Connection header received
0926 06:19:06.752121 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:06.752122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:06.752140 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:06.752225 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:19:06.752388 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:06.752961 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:06.753804 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:06.754004 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:06.754091 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:06.754251 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:19:06.754355 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:19:06.754453 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:06.754473 (+ 20us) server_negotiation.cc:299] Negotiation successful
0926 06:19:06.754492 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 06:19:06.756413 22449 raft_consensus.cc:1261] T 8ae9fb939ea14025a77b5e34516dcf78 P fce62f88fa9b4f368a39e307e8316cb4 [term 1 FOLLOWER]: Refusing update from remote peer 176dd1551f3e4f3baa6907a309ccebbb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:19:06.756433 22318 raft_consensus.cc:1261] T 8ae9fb939ea14025a77b5e34516dcf78 P 101918afe7dd499ea8f3b8c25a305d35 [term 1 FOLLOWER]: Refusing update from remote peer 176dd1551f3e4f3baa6907a309ccebbb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:19:06.756820 22650 consensus_queue.cc:1035] T 8ae9fb939ea14025a77b5e34516dcf78 P 176dd1551f3e4f3baa6907a309ccebbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fce62f88fa9b4f368a39e307e8316cb4" member_type: VOTER last_known_addr { host: "127.2.110.194" port: 35315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:19:06.756954 22645 consensus_queue.cc:1035] T 8ae9fb939ea14025a77b5e34516dcf78 P 176dd1551f3e4f3baa6907a309ccebbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "101918afe7dd499ea8f3b8c25a305d35" member_type: VOTER last_known_addr { host: "127.2.110.193" port: 46171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:19:06.759346 22653 mvcc.cc:205] Tried to move back new op lower bound from 7333492927511347200 to 7333492927344947200. Current Snapshot: MvccSnapshot[applied={T|T < 7333492927511347200}]
I20260926 06:19:06.759919 22655 mvcc.cc:205] Tried to move back new op lower bound from 7333492927511347200 to 7333492927344947200. Current Snapshot: MvccSnapshot[applied={T|T < 7333492927511347200}]
I20260926 06:19:06.760439 22654 mvcc.cc:205] Tried to move back new op lower bound from 7333492927511347200 to 7333492927344947200. Current Snapshot: MvccSnapshot[applied={T|T < 7333492927511347200 or (T in {7333492927511347200})}]
I20260926 06:19:06.761601 22176 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:19:06.761694 22176 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:19:06.762468 22176 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5225b7f40e0d49c8a3f7cef3ac47d8e9: Sending DeleteTablet for 3 replicas of tablet 8ae9fb939ea14025a77b5e34516dcf78
I20260926 06:19:06.762871 22429 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ae9fb939ea14025a77b5e34516dcf78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:19:06 UTC) from {username='slave'} at 127.0.0.1:50984
I20260926 06:19:06.762868 22298 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ae9fb939ea14025a77b5e34516dcf78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:19:06 UTC) from {username='slave'} at 127.0.0.1:44558
I20260926 06:19:06.763121 22664 tablet_replica.cc:330] T 8ae9fb939ea14025a77b5e34516dcf78 P 101918afe7dd499ea8f3b8c25a305d35: stopping tablet replica
I20260926 06:19:06.763113 22560 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ae9fb939ea14025a77b5e34516dcf78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:19:06 UTC) from {username='slave'} at 127.0.0.1:42250
I20260926 06:19:06.763208 22664 raft_consensus.cc:2229] T 8ae9fb939ea14025a77b5e34516dcf78 P 101918afe7dd499ea8f3b8c25a305d35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:19:06.763311 22664 raft_consensus.cc:2258] T 8ae9fb939ea14025a77b5e34516dcf78 P 101918afe7dd499ea8f3b8c25a305d35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:19:06.763379 22665 tablet_replica.cc:330] T 8ae9fb939ea14025a77b5e34516dcf78 P 176dd1551f3e4f3baa6907a309ccebbb: stopping tablet replica
I20260926 06:19:06.763432 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 22236
I20260926 06:19:06.763463 22665 raft_consensus.cc:2229] T 8ae9fb939ea14025a77b5e34516dcf78 P 176dd1551f3e4f3baa6907a309ccebbb [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:19:06.763553 22665 raft_consensus.cc:2258] T 8ae9fb939ea14025a77b5e34516dcf78 P 176dd1551f3e4f3baa6907a309ccebbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:19:06.763548 22663 tablet_replica.cc:330] T 8ae9fb939ea14025a77b5e34516dcf78 P fce62f88fa9b4f368a39e307e8316cb4: stopping tablet replica
I20260926 06:19:06.763635 22663 raft_consensus.cc:2229] T 8ae9fb939ea14025a77b5e34516dcf78 P fce62f88fa9b4f368a39e307e8316cb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:19:06.763726 22663 raft_consensus.cc:2258] T 8ae9fb939ea14025a77b5e34516dcf78 P fce62f88fa9b4f368a39e307e8316cb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:19:06.764074 22665 ts_tablet_manager.cc:1905] T 8ae9fb939ea14025a77b5e34516dcf78 P 176dd1551f3e4f3baa6907a309ccebbb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:19:06.765579 22663 ts_tablet_manager.cc:1905] T 8ae9fb939ea14025a77b5e34516dcf78 P fce62f88fa9b4f368a39e307e8316cb4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:19:06.765758 22665 ts_tablet_manager.cc:1918] T 8ae9fb939ea14025a77b5e34516dcf78 P 176dd1551f3e4f3baa6907a309ccebbb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:19:06.765825 22665 log.cc:1196] T 8ae9fb939ea14025a77b5e34516dcf78 P 176dd1551f3e4f3baa6907a309ccebbb: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/ts-2/wal/wals/8ae9fb939ea14025a77b5e34516dcf78
I20260926 06:19:06.766100 22665 ts_tablet_manager.cc:1939] T 8ae9fb939ea14025a77b5e34516dcf78 P 176dd1551f3e4f3baa6907a309ccebbb: Deleting consensus metadata
I20260926 06:19:06.767448 22161 catalog_manager.cc:4841] TS 176dd1551f3e4f3baa6907a309ccebbb (127.2.110.195:34345): tablet 8ae9fb939ea14025a77b5e34516dcf78 (table test-table [id=944a1d61d13344718607795c88618fb9]) successfully deleted
I20260926 06:19:06.767549 22663 ts_tablet_manager.cc:1918] T 8ae9fb939ea14025a77b5e34516dcf78 P fce62f88fa9b4f368a39e307e8316cb4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:19:06.767644 22663 log.cc:1196] T 8ae9fb939ea14025a77b5e34516dcf78 P fce62f88fa9b4f368a39e307e8316cb4: Deleting WAL directory at /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403292369990-2491-0/minicluster-data/ts-1/wal/wals/8ae9fb939ea14025a77b5e34516dcf78
I20260926 06:19:06.767954 22663 ts_tablet_manager.cc:1939] T 8ae9fb939ea14025a77b5e34516dcf78 P fce62f88fa9b4f368a39e307e8316cb4: Deleting consensus metadata
I20260926 06:19:06.768592 22162 catalog_manager.cc:4841] TS fce62f88fa9b4f368a39e307e8316cb4 (127.2.110.194:35315): tablet 8ae9fb939ea14025a77b5e34516dcf78 (table test-table [id=944a1d61d13344718607795c88618fb9]) successfully deleted
W20260926 06:19:06.772521 22160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.110.193:46171 (error 108)
I20260926 06:19:06.772702 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 22367
I20260926 06:19:06.774437 22635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:06.774254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.193:46171 (local address 127.0.0.1:44566)
0926 06:19:06.774335 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:06.774397 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.193:46171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 06:19:06.774569 22160 catalog_manager.cc:4584] TS 101918afe7dd499ea8f3b8c25a305d35 (127.2.110.193:46171): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8ae9fb939ea14025a77b5e34516dcf78: Network error: Client connection negotiation failed: client connection to 127.2.110.193:46171: connect: Connection refused (error 111)
I20260926 06:19:06.778343 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 22498
I20260926 06:19:06.783643 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 22145
2026-09-26T06:19:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (868 ms)
[----------] 6 tests from EncryptionPolicyTest (8769 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790403292369990-2491-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:19:06 dist-test-slave-hndd krb5kdc[22669](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:19:06 dist-test-slave-hndd krb5kdc[22669](info): set up 2 sockets
Sep 26 06:19:06 dist-test-slave-hndd krb5kdc[22669](info): commencing operation
krb5kdc: starting...
W20260926 06:19:07.841503 2491 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Sep 26 06:19:07 dist-test-slave-hndd krb5kdc[22669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:19:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:19:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790403292369990-2491-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790403292369990-2491-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 06:19:07.964478 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41679
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790403292369990-2491-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41679
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:19:08.048183 22682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:19:08.048430 22682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:19:08.048457 22682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:19:08.049759 22682 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:19:08.049809 22682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:19:08.049824 22682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:19:08.049837 22682 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:19:08.049853 22682 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:19:08.051440 22682 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790403292369990-2491-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41679
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790403292369990-2491-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:19:08.051723 22682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:19:08.051983 22682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:19:08.057293 22687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:08.057381 22688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:08.057611 22682 server_base.cc:1034] running on GCE node
W20260926 06:19:08.057293 22690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:08.057953 22682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:19:08.058307 22682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:19:08.059513 22682 hybrid_clock.cc:648] HybridClock initialized: now 1790403548059494 us; error 54 us; skew 500 ppm
Sep 26 06:19:08 dist-test-slave-hndd krb5kdc[22669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403548, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:19:08.062337 22682 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 06:19:08.062849 22682 webserver.cc:460] Webserver started at http://127.0.0.1:43091/ using document root <none> and password file <none>
I20260926 06:19:08.063117 22682 fs_manager.cc:359] Metadata directory not provided
I20260926 06:19:08.063189 22682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:19:08.063339 22682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:19:08.064419 22682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790403292369990-2491-0/minicluster-data/master-0/data/instance:
uuid: "1ba1aee5d3334084920887322d5e6e87"
format_stamp: "Formatted at 2026-09-26 06:19:08 on dist-test-slave-hndd"
server_key: "e49c6a578ec3c5b884fe5c7aae212701"
server_key_iv: "98e7759798c46a499d51e555956b355a"
server_key_version: "encryptionkey@0"
I20260926 06:19:08.064787 22682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790403292369990-2491-0/minicluster-data/master-0/wal/instance:
uuid: "1ba1aee5d3334084920887322d5e6e87"
format_stamp: "Formatted at 2026-09-26 06:19:08 on dist-test-slave-hndd"
server_key: "e49c6a578ec3c5b884fe5c7aae212701"
server_key_iv: "98e7759798c46a499d51e555956b355a"
server_key_version: "encryptionkey@0"
I20260926 06:19:08.066277 22682 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 06:19:08.067106 22697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:08.067332 22682 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:19:08.067442 22682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "1ba1aee5d3334084920887322d5e6e87"
format_stamp: "Formatted at 2026-09-26 06:19:08 on dist-test-slave-hndd"
server_key: "e49c6a578ec3c5b884fe5c7aae212701"
server_key_iv: "98e7759798c46a499d51e555956b355a"
server_key_version: "encryptionkey@0"
I20260926 06:19:08.067519 22682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:19:08.078773 22682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:19:08.081171 22682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:19:08.081439 22682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:19:08.085505 22682 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41679
I20260926 06:19:08.085546 22749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41679 every 8 connection(s)
I20260926 06:19:08.085933 22682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:19:08.086839 22750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:19:08.089182 22750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ba1aee5d3334084920887322d5e6e87: Bootstrap starting.
I20260926 06:19:08.089900 22750 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ba1aee5d3334084920887322d5e6e87: Neither blocks nor log segments found. Creating new log.
I20260926 06:19:08.089941 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 22682
I20260926 06:19:08.090052 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790403292369990-2491-0/minicluster-data/master-0/wal/instance
I20260926 06:19:08.090253 22750 log.cc:824] T 00000000000000000000000000000000 P 1ba1aee5d3334084920887322d5e6e87: Log is configured to *not* fsync() on all Append() calls
I20260926 06:19:08.090289 2491 external_mini_cluster.cc:1428] Setting key ceb6407da4e9ef92aed47650840b0d2b
I20260926 06:19:08.091073 22750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ba1aee5d3334084920887322d5e6e87: No bootstrap required, opened a new log
I20260926 06:19:08.093848 22750 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ba1aee5d3334084920887322d5e6e87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba1aee5d3334084920887322d5e6e87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41679 } }
I20260926 06:19:08.094237 22750 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ba1aee5d3334084920887322d5e6e87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:19:08.094300 22750 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ba1aee5d3334084920887322d5e6e87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ba1aee5d3334084920887322d5e6e87, State: Initialized, Role: FOLLOWER
I20260926 06:19:08.094483 22750 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ba1aee5d3334084920887322d5e6e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba1aee5d3334084920887322d5e6e87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41679 } }
I20260926 06:19:08.094621 22750 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ba1aee5d3334084920887322d5e6e87 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:19:08.094663 22750 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ba1aee5d3334084920887322d5e6e87 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:19:08.094720 22750 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ba1aee5d3334084920887322d5e6e87 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:19:08.095808 22750 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ba1aee5d3334084920887322d5e6e87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba1aee5d3334084920887322d5e6e87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41679 } }
I20260926 06:19:08.095956 22750 leader_election.cc:302] T 00000000000000000000000000000000 P 1ba1aee5d3334084920887322d5e6e87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ba1aee5d3334084920887322d5e6e87; no voters:
I20260926 06:19:08.096199 22750 leader_election.cc:288] T 00000000000000000000000000000000 P 1ba1aee5d3334084920887322d5e6e87 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:19:08.096223 22755 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ba1aee5d3334084920887322d5e6e87 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:19:08.096351 22755 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ba1aee5d3334084920887322d5e6e87 [term 1 LEADER]: Becoming Leader. State: Replica: 1ba1aee5d3334084920887322d5e6e87, State: Running, Role: LEADER
I20260926 06:19:08.096446 22755 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ba1aee5d3334084920887322d5e6e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba1aee5d3334084920887322d5e6e87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41679 } }
I20260926 06:19:08.096496 22750 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ba1aee5d3334084920887322d5e6e87 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:19:08.096769 22756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ba1aee5d3334084920887322d5e6e87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ba1aee5d3334084920887322d5e6e87. Latest consensus state: current_term: 1 leader_uuid: "1ba1aee5d3334084920887322d5e6e87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba1aee5d3334084920887322d5e6e87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41679 } } }
I20260926 06:19:08.097030 22756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ba1aee5d3334084920887322d5e6e87 [sys.catalog]: This master's current role is: LEADER
I20260926 06:19:08.096947 22757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ba1aee5d3334084920887322d5e6e87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ba1aee5d3334084920887322d5e6e87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba1aee5d3334084920887322d5e6e87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41679 } } }
I20260926 06:19:08.097667 22757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ba1aee5d3334084920887322d5e6e87 [sys.catalog]: This master's current role is: LEADER
I20260926 06:19:08.097843 22763 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:19:08.098382 22763 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:19:08.100200 22763 catalog_manager.cc:1347] Generated new cluster ID: 0157320980da4e2d9960cadd94d24334
I20260926 06:19:08.100247 22763 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 06:19:08 dist-test-slave-hndd krb5kdc[22669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 06:19:08.104640 22754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:08.091082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35599 (local address 127.0.0.1:41679)
0926 06:19:08.091256 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:08.091265 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:08.091335 (+ 70us) server_negotiation.cc:407] Connection header received
0926 06:19:08.092661 (+ 1326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:08.092671 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:08.092738 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:08.092900 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:08.093750 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:08.095963 (+ 2213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:08.097049 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:08.097686 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:08.097777 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:08.100970 (+ 3193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:08.100973 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:08.100974 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:19:08.100986 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:08.102706 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:08.103139 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:08.103142 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:08.103142 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:08.103348 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:08.103641 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:08.103642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:08.103642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:08.103919 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:08.104196 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:08.104294 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 06:19:08.104429 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":74,"threads_started":1}
I20260926 06:19:08.105882 2491 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:19:08.110883 22763 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:19:08.111609 22763 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:19:08.121941 22754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:08.107381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39681 (local address 127.0.0.1:41679)
0926 06:19:08.107428 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:08.107430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:08.109618 (+ 2188us) server_negotiation.cc:407] Connection header received
0926 06:19:08.109702 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:08.109703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:08.109714 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:08.109784 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:08.110916 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:08.111459 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:08.112308 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:08.112582 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:08.112692 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:08.119209 (+ 6517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:08.119211 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:08.119211 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:19:08.119220 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:08.120667 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:08.121076 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:08.121077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:08.121078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:08.121136 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:08.121398 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:08.121399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:08.121399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:08.121519 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:08.121635 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:08.121754 (+ 119us) server_negotiation.cc:299] Negotiation successful
0926 06:19:08.121820 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 06:19:08.124926 22763 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ba1aee5d3334084920887322d5e6e87: Generated new TSK 0
I20260926 06:19:08.125149 22763 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:19:08.130080 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 22682
W20260926 06:19:08.134728 22667 connection.cc:535] client connection to 127.0.0.1:41679 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41679): Connection reset by peer (error 104)
2026-09-26T06:19:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1344 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790403292369990-2491-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:19:08 dist-test-slave-hndd krb5kdc[22781](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:19:08 dist-test-slave-hndd krb5kdc[22781](info): set up 2 sockets
Sep 26 06:19:08 dist-test-slave-hndd krb5kdc[22781](info): commencing operation
krb5kdc: starting...
W20260926 06:19:09.217708 2491 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Sep 26 06:19:09 dist-test-slave-hndd krb5kdc[22781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:19:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:19:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790403292369990-2491-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790403292369990-2491-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 06:19:09.345161 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39675
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790403292369990-2491-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:19:09.429844 22794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:19:09.430115 22794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:19:09.430160 22794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:19:09.431406 22794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:19:09.431489 22794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:19:09.431516 22794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:19:09.431547 22794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:19:09.431577 22794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:19:09.433163 22794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790403292369990-2491-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39675
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790403292369990-2491-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:19:09.433493 22794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:19:09.433748 22794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:19:09.438820 22802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:09.438867 22800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:09.438839 22799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:09.438865 22794 server_base.cc:1034] running on GCE node
I20260926 06:19:09.439239 22794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:19:09.439584 22794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:19:09.440742 22794 hybrid_clock.cc:648] HybridClock initialized: now 1790403549440721 us; error 41 us; skew 500 ppm
Sep 26 06:19:09 dist-test-slave-hndd krb5kdc[22781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403549, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:19:09.443430 22794 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 06:19:09.443918 22794 webserver.cc:460] Webserver started at http://127.0.0.1:38085/ using document root <none> and password file <none>
I20260926 06:19:09.444130 22794 fs_manager.cc:359] Metadata directory not provided
I20260926 06:19:09.444170 22794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:19:09.444259 22794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:19:09.445282 22794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790403292369990-2491-0/minicluster-data/master-0/data/instance:
uuid: "ff15f25501514c2cb749bfa837a317dd"
format_stamp: "Formatted at 2026-09-26 06:19:09 on dist-test-slave-hndd"
server_key: "f9f5b035512ea45ff30be2c358e1777c"
server_key_iv: "6e45ad4eba57d2c887dd2b3213aaa747"
server_key_version: "encryptionkey@0"
I20260926 06:19:09.445679 22794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790403292369990-2491-0/minicluster-data/master-0/wal/instance:
uuid: "ff15f25501514c2cb749bfa837a317dd"
format_stamp: "Formatted at 2026-09-26 06:19:09 on dist-test-slave-hndd"
server_key: "f9f5b035512ea45ff30be2c358e1777c"
server_key_iv: "6e45ad4eba57d2c887dd2b3213aaa747"
server_key_version: "encryptionkey@0"
I20260926 06:19:09.447005 22794 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:19:09.447763 22809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:09.448011 22794 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:19:09.448086 22794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "ff15f25501514c2cb749bfa837a317dd"
format_stamp: "Formatted at 2026-09-26 06:19:09 on dist-test-slave-hndd"
server_key: "f9f5b035512ea45ff30be2c358e1777c"
server_key_iv: "6e45ad4eba57d2c887dd2b3213aaa747"
server_key_version: "encryptionkey@0"
I20260926 06:19:09.448182 22794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:19:09.462265 22794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:19:09.464578 22794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:19:09.464824 22794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:19:09.470315 22794 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39675
I20260926 06:19:09.470335 22861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39675 every 8 connection(s)
I20260926 06:19:09.470894 22794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:19:09.471853 22862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:19:09.474325 22862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff15f25501514c2cb749bfa837a317dd: Bootstrap starting.
I20260926 06:19:09.474938 22862 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff15f25501514c2cb749bfa837a317dd: Neither blocks nor log segments found. Creating new log.
I20260926 06:19:09.475373 22862 log.cc:824] T 00000000000000000000000000000000 P ff15f25501514c2cb749bfa837a317dd: Log is configured to *not* fsync() on all Append() calls
I20260926 06:19:09.476171 22862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff15f25501514c2cb749bfa837a317dd: No bootstrap required, opened a new log
I20260926 06:19:09.477919 22862 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff15f25501514c2cb749bfa837a317dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff15f25501514c2cb749bfa837a317dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39675 } }
I20260926 06:19:09.478093 22862 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff15f25501514c2cb749bfa837a317dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:19:09.478142 22862 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff15f25501514c2cb749bfa837a317dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff15f25501514c2cb749bfa837a317dd, State: Initialized, Role: FOLLOWER
I20260926 06:19:09.478338 22862 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff15f25501514c2cb749bfa837a317dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff15f25501514c2cb749bfa837a317dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39675 } }
I20260926 06:19:09.478443 22862 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff15f25501514c2cb749bfa837a317dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:19:09.478489 22862 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff15f25501514c2cb749bfa837a317dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:19:09.478538 22862 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff15f25501514c2cb749bfa837a317dd [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:19:09.479125 22862 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff15f25501514c2cb749bfa837a317dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff15f25501514c2cb749bfa837a317dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39675 } }
I20260926 06:19:09.479250 22862 leader_election.cc:302] T 00000000000000000000000000000000 P ff15f25501514c2cb749bfa837a317dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff15f25501514c2cb749bfa837a317dd; no voters:
I20260926 06:19:09.479447 22862 leader_election.cc:288] T 00000000000000000000000000000000 P ff15f25501514c2cb749bfa837a317dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:19:09.479501 22865 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff15f25501514c2cb749bfa837a317dd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:19:09.479763 22862 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff15f25501514c2cb749bfa837a317dd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:19:09.479765 22865 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff15f25501514c2cb749bfa837a317dd [term 1 LEADER]: Becoming Leader. State: Replica: ff15f25501514c2cb749bfa837a317dd, State: Running, Role: LEADER
I20260926 06:19:09.479947 22865 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff15f25501514c2cb749bfa837a317dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff15f25501514c2cb749bfa837a317dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39675 } }
I20260926 06:19:09.480521 22867 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff15f25501514c2cb749bfa837a317dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff15f25501514c2cb749bfa837a317dd. Latest consensus state: current_term: 1 leader_uuid: "ff15f25501514c2cb749bfa837a317dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff15f25501514c2cb749bfa837a317dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39675 } } }
I20260926 06:19:09.480733 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 22794
I20260926 06:19:09.480842 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790403292369990-2491-0/minicluster-data/master-0/wal/instance
I20260926 06:19:09.480939 22867 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff15f25501514c2cb749bfa837a317dd [sys.catalog]: This master's current role is: LEADER
I20260926 06:19:09.480846 22866 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff15f25501514c2cb749bfa837a317dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff15f25501514c2cb749bfa837a317dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff15f25501514c2cb749bfa837a317dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39675 } } }
I20260926 06:19:09.481068 2491 external_mini_cluster.cc:1428] Setting key d3df9a1f7b048e75d921c8e972cb5d56
I20260926 06:19:09.481088 22866 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff15f25501514c2cb749bfa837a317dd [sys.catalog]: This master's current role is: LEADER
I20260926 06:19:09.481235 22878 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:19:09.482061 22878 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:19:09.484885 22878 catalog_manager.cc:1347] Generated new cluster ID: 1d6c426d4a01462ca78567020d99f7a0
I20260926 06:19:09.484947 22878 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:19:09.487711 22883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:09.481606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49593 (local address 127.0.0.1:39675)
0926 06:19:09.481821 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:09.481831 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:09.481868 (+ 37us) server_negotiation.cc:407] Connection header received
0926 06:19:09.481997 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:09.482005 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:09.482065 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:09.482223 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:09.482433 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:09.483854 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:09.485443 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:09.486099 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:09.486219 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:09.487002 (+ 783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:09.487006 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:09.487007 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:19:09.487220 (+ 213us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:09.487261 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:19:09.487277 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:09.487375 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:09.487452 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 06:19:09.487482 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":115,"threads_started":1}
I20260926 06:19:09.488644 2491 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:19:09.497159 22883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:09.490113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41437 (local address 127.0.0.1:39675)
0926 06:19:09.490147 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:09.490149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:09.493565 (+ 3416us) server_negotiation.cc:407] Connection header received
0926 06:19:09.493636 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:09.493637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:09.493650 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:09.493712 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:09.493847 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:09.494600 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:09.495640 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:09.496009 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:09.496088 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:09.496880 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:09.496882 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:09.496882 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:19:09.496888 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:09.496908 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:19:09.496913 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:09.496980 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:09.497050 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 06:19:09.497066 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 06:19:09.506587 22878 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:19:09.507166 22878 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:19:09.511855 22878 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff15f25501514c2cb749bfa837a317dd: Generated new TSK 0
I20260926 06:19:09.512089 22878 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:19:09.518083 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 22794
W20260926 06:19:09.522207 22779 connection.cc:535] client connection to 127.0.0.1:39675 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39675): Connection reset by peer (error 104)
2026-09-26T06:19:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1387 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790403292369990-2491-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:19:09 dist-test-slave-hndd krb5kdc[22893](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:19:09 dist-test-slave-hndd krb5kdc[22893](info): set up 2 sockets
Sep 26 06:19:09 dist-test-slave-hndd krb5kdc[22893](info): commencing operation
krb5kdc: starting...
W20260926 06:19:10.573477 2491 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Sep 26 06:19:10 dist-test-slave-hndd krb5kdc[22893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:19:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:19:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790403292369990-2491-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790403292369990-2491-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 06:19:10.706712 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36385
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790403292369990-2491-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36385
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:19:10.792244 22906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:19:10.792479 22906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:19:10.792505 22906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:19:10.793793 22906 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:19:10.793840 22906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:19:10.793855 22906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:19:10.793869 22906 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:19:10.793881 22906 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:19:10.795509 22906 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790403292369990-2491-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36385
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790403292369990-2491-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:19:10.795801 22906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:19:10.796046 22906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:19:10.801116 22911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:10.801116 22912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:10.801373 22914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:10.801436 22906 server_base.cc:1034] running on GCE node
I20260926 06:19:10.801851 22906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:19:10.802244 22906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:19:10.803445 22906 hybrid_clock.cc:648] HybridClock initialized: now 1790403550803414 us; error 53 us; skew 500 ppm
Sep 26 06:19:10 dist-test-slave-hndd krb5kdc[22893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:19:10.806416 22906 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 06:19:10.806898 22906 webserver.cc:460] Webserver started at http://127.0.0.1:41883/ using document root <none> and password file <none>
I20260926 06:19:10.807128 22906 fs_manager.cc:359] Metadata directory not provided
I20260926 06:19:10.807190 22906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:19:10.807294 22906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:19:10.808427 22906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790403292369990-2491-0/minicluster-data/master-0/data/instance:
uuid: "ea0b0fa7c0ee410998e77702e9b77196"
format_stamp: "Formatted at 2026-09-26 06:19:10 on dist-test-slave-hndd"
server_key: "c8f4ebfe466a7a1d8d18cb40ff712649"
server_key_iv: "a83b96c26921fdf055684a1b83e1c8b9"
server_key_version: "encryptionkey@0"
I20260926 06:19:10.808775 22906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790403292369990-2491-0/minicluster-data/master-0/wal/instance:
uuid: "ea0b0fa7c0ee410998e77702e9b77196"
format_stamp: "Formatted at 2026-09-26 06:19:10 on dist-test-slave-hndd"
server_key: "c8f4ebfe466a7a1d8d18cb40ff712649"
server_key_iv: "a83b96c26921fdf055684a1b83e1c8b9"
server_key_version: "encryptionkey@0"
I20260926 06:19:10.810307 22906 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 06:19:10.811151 22921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:10.811364 22906 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:19:10.811460 22906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "ea0b0fa7c0ee410998e77702e9b77196"
format_stamp: "Formatted at 2026-09-26 06:19:10 on dist-test-slave-hndd"
server_key: "c8f4ebfe466a7a1d8d18cb40ff712649"
server_key_iv: "a83b96c26921fdf055684a1b83e1c8b9"
server_key_version: "encryptionkey@0"
I20260926 06:19:10.811519 22906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:19:10.814955 22906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:19:10.816952 22906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:19:10.817068 22906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:19:10.820935 22906 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36385
I20260926 06:19:10.821012 22973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36385 every 8 connection(s)
I20260926 06:19:10.821411 22906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:19:10.822099 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 22906
I20260926 06:19:10.822227 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790403292369990-2491-0/minicluster-data/master-0/wal/instance
I20260926 06:19:10.822435 2491 external_mini_cluster.cc:1428] Setting key e2dec1d46c405037a732e16ad55b0c63
I20260926 06:19:10.822510 22974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:19:10.825106 22977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:10.823007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56265 (local address 127.0.0.1:36385)
0926 06:19:10.823211 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:10.823220 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:10.823256 (+ 36us) server_negotiation.cc:407] Connection header received
0926 06:19:10.823708 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:10.823718 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:10.823749 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:10.823914 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:10.824566 (+ 652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:10.824570 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:10.824570 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:19:10.824575 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:10.824623 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:19:10.824637 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:10.824719 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:10.824757 (+ 38us) server_negotiation.cc:299] Negotiation successful
0926 06:19:10.824918 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":85,"threads_started":1}
I20260926 06:19:10.826040 22974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea0b0fa7c0ee410998e77702e9b77196: Bootstrap starting.
I20260926 06:19:10.826263 2491 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:19:10.826716 22974 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea0b0fa7c0ee410998e77702e9b77196: Neither blocks nor log segments found. Creating new log.
I20260926 06:19:10.827484 22974 log.cc:824] T 00000000000000000000000000000000 P ea0b0fa7c0ee410998e77702e9b77196: Log is configured to *not* fsync() on all Append() calls
I20260926 06:19:10.828500 22974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea0b0fa7c0ee410998e77702e9b77196: No bootstrap required, opened a new log
I20260926 06:19:10.829118 22977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:10.827750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41557 (local address 127.0.0.1:36385)
0926 06:19:10.827806 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:10.827808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:10.828065 (+ 257us) server_negotiation.cc:407] Connection header received
0926 06:19:10.828147 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:10.828148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:10.828154 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:10.828232 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:10.828875 (+ 643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:10.828876 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:10.828876 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:19:10.828880 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:10.828895 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:19:10.828899 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:10.828968 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:10.829031 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 06:19:10.829046 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 06:19:10.830760 22974 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea0b0fa7c0ee410998e77702e9b77196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea0b0fa7c0ee410998e77702e9b77196" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36385 } }
I20260926 06:19:10.830971 22974 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea0b0fa7c0ee410998e77702e9b77196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:19:10.831015 22974 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea0b0fa7c0ee410998e77702e9b77196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea0b0fa7c0ee410998e77702e9b77196, State: Initialized, Role: FOLLOWER
I20260926 06:19:10.831162 22974 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea0b0fa7c0ee410998e77702e9b77196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea0b0fa7c0ee410998e77702e9b77196" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36385 } }
I20260926 06:19:10.831281 22974 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea0b0fa7c0ee410998e77702e9b77196 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:19:10.831315 22974 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea0b0fa7c0ee410998e77702e9b77196 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:19:10.831364 22974 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea0b0fa7c0ee410998e77702e9b77196 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:19:10.831929 22974 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea0b0fa7c0ee410998e77702e9b77196 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea0b0fa7c0ee410998e77702e9b77196" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36385 } }
I20260926 06:19:10.832016 22974 leader_election.cc:302] T 00000000000000000000000000000000 P ea0b0fa7c0ee410998e77702e9b77196 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea0b0fa7c0ee410998e77702e9b77196; no voters:
I20260926 06:19:10.832329 22974 leader_election.cc:288] T 00000000000000000000000000000000 P ea0b0fa7c0ee410998e77702e9b77196 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:19:10.832413 22984 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea0b0fa7c0ee410998e77702e9b77196 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:19:10.832659 22984 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea0b0fa7c0ee410998e77702e9b77196 [term 1 LEADER]: Becoming Leader. State: Replica: ea0b0fa7c0ee410998e77702e9b77196, State: Running, Role: LEADER
I20260926 06:19:10.832762 22984 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea0b0fa7c0ee410998e77702e9b77196 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea0b0fa7c0ee410998e77702e9b77196" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36385 } }
I20260926 06:19:10.832837 22974 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea0b0fa7c0ee410998e77702e9b77196 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:19:10.833164 22986 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea0b0fa7c0ee410998e77702e9b77196 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea0b0fa7c0ee410998e77702e9b77196. Latest consensus state: current_term: 1 leader_uuid: "ea0b0fa7c0ee410998e77702e9b77196" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea0b0fa7c0ee410998e77702e9b77196" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36385 } } }
I20260926 06:19:10.833250 22986 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea0b0fa7c0ee410998e77702e9b77196 [sys.catalog]: This master's current role is: LEADER
I20260926 06:19:10.833426 22985 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea0b0fa7c0ee410998e77702e9b77196 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea0b0fa7c0ee410998e77702e9b77196" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea0b0fa7c0ee410998e77702e9b77196" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36385 } } }
I20260926 06:19:10.833511 22985 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea0b0fa7c0ee410998e77702e9b77196 [sys.catalog]: This master's current role is: LEADER
I20260926 06:19:10.833792 22989 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:19:10.834384 22989 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:19:10.835985 22989 catalog_manager.cc:1347] Generated new cluster ID: ff72538c68e4427ab14eef97454248a2
I20260926 06:19:10.836048 22989 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:19:10.854965 22989 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:19:10.855288 22989 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:19:10.863924 22989 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea0b0fa7c0ee410998e77702e9b77196: Generated new TSK 0
I20260926 06:19:10.864167 22989 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:19:10.873143 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 22906
2026-09-26T06:19:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1355 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790403292369990-2491-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:19:10 dist-test-slave-hndd krb5kdc[23005](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:19:10 dist-test-slave-hndd krb5kdc[23005](info): set up 2 sockets
Sep 26 06:19:10 dist-test-slave-hndd krb5kdc[23005](info): commencing operation
krb5kdc: starting...
W20260926 06:19:11.937556 2491 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Sep 26 06:19:12 dist-test-slave-hndd krb5kdc[23005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:19:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:19:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790403292369990-2491-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790403292369990-2491-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 06:19:12.060246 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46583
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790403292369990-2491-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46583
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:19:12.144490 23018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:19:12.144766 23018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:19:12.144810 23018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:19:12.146031 23018 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:19:12.146078 23018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:19:12.146093 23018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:19:12.146106 23018 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:19:12.146118 23018 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:19:12.147547 23018 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790403292369990-2491-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790403292369990-2491-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:19:12.147753 23018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:19:12.147943 23018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:19:12.153095 23026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:12.153080 23024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:12.153105 23018 server_base.cc:1034] running on GCE node
W20260926 06:19:12.153110 23023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:12.153652 23018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:19:12.153973 23018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:19:12.155162 23018 hybrid_clock.cc:648] HybridClock initialized: now 1790403552155159 us; error 41 us; skew 500 ppm
Sep 26 06:19:12 dist-test-slave-hndd krb5kdc[23005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403552, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:19:12.158095 23018 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 06:19:12.158614 23018 webserver.cc:460] Webserver started at http://127.0.0.1:34839/ using document root <none> and password file <none>
I20260926 06:19:12.158897 23018 fs_manager.cc:359] Metadata directory not provided
I20260926 06:19:12.158950 23018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:19:12.159134 23018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:19:12.160370 23018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790403292369990-2491-0/minicluster-data/master-0/data/instance:
uuid: "4fd7038f9ba04ad7a454d156389008f0"
format_stamp: "Formatted at 2026-09-26 06:19:12 on dist-test-slave-hndd"
server_key: "9075736baf2f3aa7758e671feb560c8e"
server_key_iv: "634f0166156f37b88e631695c7ea63f7"
server_key_version: "encryptionkey@0"
I20260926 06:19:12.160797 23018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790403292369990-2491-0/minicluster-data/master-0/wal/instance:
uuid: "4fd7038f9ba04ad7a454d156389008f0"
format_stamp: "Formatted at 2026-09-26 06:19:12 on dist-test-slave-hndd"
server_key: "9075736baf2f3aa7758e671feb560c8e"
server_key_iv: "634f0166156f37b88e631695c7ea63f7"
server_key_version: "encryptionkey@0"
I20260926 06:19:12.162388 23018 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:19:12.163329 23033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:12.163527 23018 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:19:12.163622 23018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "4fd7038f9ba04ad7a454d156389008f0"
format_stamp: "Formatted at 2026-09-26 06:19:12 on dist-test-slave-hndd"
server_key: "9075736baf2f3aa7758e671feb560c8e"
server_key_iv: "634f0166156f37b88e631695c7ea63f7"
server_key_version: "encryptionkey@0"
I20260926 06:19:12.163679 23018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:19:12.177091 23018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:19:12.179388 23018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:19:12.179514 23018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:19:12.184919 23018 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46583
I20260926 06:19:12.184945 23085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46583 every 8 connection(s)
I20260926 06:19:12.185453 23018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:19:12.186524 23086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:19:12.188831 23086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fd7038f9ba04ad7a454d156389008f0: Bootstrap starting.
I20260926 06:19:12.189508 23086 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fd7038f9ba04ad7a454d156389008f0: Neither blocks nor log segments found. Creating new log.
I20260926 06:19:12.189796 23086 log.cc:824] T 00000000000000000000000000000000 P 4fd7038f9ba04ad7a454d156389008f0: Log is configured to *not* fsync() on all Append() calls
I20260926 06:19:12.190567 23086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fd7038f9ba04ad7a454d156389008f0: No bootstrap required, opened a new log
I20260926 06:19:12.192231 23086 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fd7038f9ba04ad7a454d156389008f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fd7038f9ba04ad7a454d156389008f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46583 } }
I20260926 06:19:12.192374 23086 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4fd7038f9ba04ad7a454d156389008f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:19:12.192397 23086 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fd7038f9ba04ad7a454d156389008f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd7038f9ba04ad7a454d156389008f0, State: Initialized, Role: FOLLOWER
I20260926 06:19:12.192561 23086 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fd7038f9ba04ad7a454d156389008f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fd7038f9ba04ad7a454d156389008f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46583 } }
I20260926 06:19:12.192627 23086 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4fd7038f9ba04ad7a454d156389008f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:19:12.192649 23086 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fd7038f9ba04ad7a454d156389008f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:19:12.192718 23086 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fd7038f9ba04ad7a454d156389008f0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:19:12.193313 23086 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fd7038f9ba04ad7a454d156389008f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fd7038f9ba04ad7a454d156389008f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46583 } }
I20260926 06:19:12.193491 23086 leader_election.cc:302] T 00000000000000000000000000000000 P 4fd7038f9ba04ad7a454d156389008f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fd7038f9ba04ad7a454d156389008f0; no voters:
I20260926 06:19:12.193697 23086 leader_election.cc:288] T 00000000000000000000000000000000 P 4fd7038f9ba04ad7a454d156389008f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:19:12.193743 23089 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fd7038f9ba04ad7a454d156389008f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:19:12.193933 23089 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4fd7038f9ba04ad7a454d156389008f0 [term 1 LEADER]: Becoming Leader. State: Replica: 4fd7038f9ba04ad7a454d156389008f0, State: Running, Role: LEADER
I20260926 06:19:12.193987 23086 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fd7038f9ba04ad7a454d156389008f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:19:12.194106 23089 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fd7038f9ba04ad7a454d156389008f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fd7038f9ba04ad7a454d156389008f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46583 } }
I20260926 06:19:12.194541 23090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fd7038f9ba04ad7a454d156389008f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4fd7038f9ba04ad7a454d156389008f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fd7038f9ba04ad7a454d156389008f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46583 } } }
I20260926 06:19:12.194550 23091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fd7038f9ba04ad7a454d156389008f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fd7038f9ba04ad7a454d156389008f0. Latest consensus state: current_term: 1 leader_uuid: "4fd7038f9ba04ad7a454d156389008f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fd7038f9ba04ad7a454d156389008f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46583 } } }
I20260926 06:19:12.194644 23090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fd7038f9ba04ad7a454d156389008f0 [sys.catalog]: This master's current role is: LEADER
I20260926 06:19:12.194672 23091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fd7038f9ba04ad7a454d156389008f0 [sys.catalog]: This master's current role is: LEADER
I20260926 06:19:12.194875 23093 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:19:12.195459 23093 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:19:12.195660 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 23018
I20260926 06:19:12.195797 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790403292369990-2491-0/minicluster-data/master-0/wal/instance
I20260926 06:19:12.196013 2491 external_mini_cluster.cc:1428] Setting key ba5f59418505108d5fa44d35c17c26a4
I20260926 06:19:12.197686 23093 catalog_manager.cc:1347] Generated new cluster ID: 907df3e3ce764467be9bb2365e3cf7a3
I20260926 06:19:12.197750 23093 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 06:19:12 dist-test-slave-hndd krb5kdc[23005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 06:19:12.208854 23093 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:19:12.209666 23093 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:19:12.211623 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:12.196754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40765 (local address 127.0.0.1:46583)
0926 06:19:12.197119 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:12.197128 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:12.197264 (+ 136us) server_negotiation.cc:407] Connection header received
0926 06:19:12.197444 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:12.197452 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:12.197511 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:12.201569 (+ 4058us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:12.202246 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:12.203329 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:12.204089 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:12.204560 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:12.204700 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:12.207164 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:12.207168 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:12.207169 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:19:12.207181 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:12.209966 (+ 2785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:12.210376 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:12.210379 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:12.210380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:12.210481 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:12.210712 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:12.210713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:12.210713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:12.210957 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:12.211127 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:12.211246 (+ 119us) server_negotiation.cc:299] Negotiation successful
0926 06:19:12.211416 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":77,"threads_started":1}
I20260926 06:19:12.212592 2491 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:19:12.224352 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:12.214060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58785 (local address 127.0.0.1:46583)
0926 06:19:12.214110 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:12.214112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:12.217610 (+ 3498us) server_negotiation.cc:407] Connection header received
0926 06:19:12.217711 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:12.217712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:12.217724 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:12.217781 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:12.218378 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:12.218911 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:12.219686 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:12.220236 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:12.220325 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:12.221737 (+ 1412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:12.221738 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:12.221739 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:19:12.221746 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:12.223181 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:12.223550 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:12.223552 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:12.223552 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:12.223612 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:12.223825 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:12.223826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:12.223827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:12.223940 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:12.224064 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:12.224149 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 06:19:12.224241 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 06:19:12.228183 23093 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4fd7038f9ba04ad7a454d156389008f0: Generated new TSK 0
I20260926 06:19:12.228391 23093 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:19:12.231418 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 23018
2026-09-26T06:19:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1358 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790403292369990-2491-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:19:12 dist-test-slave-hndd krb5kdc[23117](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:19:12 dist-test-slave-hndd krb5kdc[23117](info): set up 2 sockets
Sep 26 06:19:12 dist-test-slave-hndd krb5kdc[23117](info): commencing operation
krb5kdc: starting...
W20260926 06:19:13.289685 2491 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Sep 26 06:19:13 dist-test-slave-hndd krb5kdc[23117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:19:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:19:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790403292369990-2491-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790403292369990-2491-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 06:19:13.420490 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33637
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790403292369990-2491-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33637
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:19:13.507485 23130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:19:13.507763 23130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:19:13.507807 23130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:19:13.509188 23130 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:19:13.509272 23130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:19:13.509296 23130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:19:13.509318 23130 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:19:13.509382 23130 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:19:13.510993 23130 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790403292369990-2491-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33637
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790403292369990-2491-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:19:13.511303 23130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:19:13.511559 23130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:19:13.517302 23136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:13.517297 23138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:13.517297 23135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:13.517329 23130 server_base.cc:1034] running on GCE node
I20260926 06:19:13.517851 23130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:19:13.518172 23130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:19:13.519407 23130 hybrid_clock.cc:648] HybridClock initialized: now 1790403553519345 us; error 98 us; skew 500 ppm
Sep 26 06:19:13 dist-test-slave-hndd krb5kdc[23117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403553, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:19:13.526856 23130 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 06:19:13.527387 23130 webserver.cc:460] Webserver started at http://127.0.0.1:43793/ using document root <none> and password file <none>
I20260926 06:19:13.527590 23130 fs_manager.cc:359] Metadata directory not provided
I20260926 06:19:13.527634 23130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:19:13.527730 23130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:19:13.528788 23130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790403292369990-2491-0/minicluster-data/master-0/data/instance:
uuid: "1fa69357231b45588341634941318dc0"
format_stamp: "Formatted at 2026-09-26 06:19:13 on dist-test-slave-hndd"
server_key: "233c95ae8dadd7da4542ee187570d1d7"
server_key_iv: "a0209e40da79e444dfef82e320f59744"
server_key_version: "encryptionkey@0"
I20260926 06:19:13.529105 23130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790403292369990-2491-0/minicluster-data/master-0/wal/instance:
uuid: "1fa69357231b45588341634941318dc0"
format_stamp: "Formatted at 2026-09-26 06:19:13 on dist-test-slave-hndd"
server_key: "233c95ae8dadd7da4542ee187570d1d7"
server_key_iv: "a0209e40da79e444dfef82e320f59744"
server_key_version: "encryptionkey@0"
I20260926 06:19:13.530718 23130 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:19:13.531992 23145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:13.532394 23130 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:19:13.532624 23130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "1fa69357231b45588341634941318dc0"
format_stamp: "Formatted at 2026-09-26 06:19:13 on dist-test-slave-hndd"
server_key: "233c95ae8dadd7da4542ee187570d1d7"
server_key_iv: "a0209e40da79e444dfef82e320f59744"
server_key_version: "encryptionkey@0"
I20260926 06:19:13.532721 23130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:19:13.553452 23130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:19:13.555727 23130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:19:13.555848 23130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:19:13.559845 23130 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33637
I20260926 06:19:13.559913 23197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33637 every 8 connection(s)
I20260926 06:19:13.560245 23130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:19:13.561290 23198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:19:13.564141 23198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fa69357231b45588341634941318dc0: Bootstrap starting.
I20260926 06:19:13.564761 23198 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1fa69357231b45588341634941318dc0: Neither blocks nor log segments found. Creating new log.
I20260926 06:19:13.565054 23198 log.cc:824] T 00000000000000000000000000000000 P 1fa69357231b45588341634941318dc0: Log is configured to *not* fsync() on all Append() calls
I20260926 06:19:13.565861 23198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fa69357231b45588341634941318dc0: No bootstrap required, opened a new log
I20260926 06:19:13.566205 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 23130
I20260926 06:19:13.566345 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790403292369990-2491-0/minicluster-data/master-0/wal/instance
I20260926 06:19:13.566560 2491 external_mini_cluster.cc:1428] Setting key 0916bf84a787fdf06f68c4325f5afbfd
I20260926 06:19:13.567934 23198 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1fa69357231b45588341634941318dc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fa69357231b45588341634941318dc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33637 } }
I20260926 06:19:13.568100 23198 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1fa69357231b45588341634941318dc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:19:13.568145 23198 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1fa69357231b45588341634941318dc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fa69357231b45588341634941318dc0, State: Initialized, Role: FOLLOWER
I20260926 06:19:13.568300 23198 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fa69357231b45588341634941318dc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fa69357231b45588341634941318dc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33637 } }
I20260926 06:19:13.568401 23198 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1fa69357231b45588341634941318dc0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:19:13.568446 23198 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1fa69357231b45588341634941318dc0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:19:13.568500 23198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1fa69357231b45588341634941318dc0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:19:13.569299 23198 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1fa69357231b45588341634941318dc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fa69357231b45588341634941318dc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33637 } }
I20260926 06:19:13.569463 23198 leader_election.cc:302] T 00000000000000000000000000000000 P 1fa69357231b45588341634941318dc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1fa69357231b45588341634941318dc0; no voters:
I20260926 06:19:13.569653 23198 leader_election.cc:288] T 00000000000000000000000000000000 P 1fa69357231b45588341634941318dc0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:19:13.569767 23203 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1fa69357231b45588341634941318dc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:19:13.569896 23198 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fa69357231b45588341634941318dc0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:19:13.570601 23203 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1fa69357231b45588341634941318dc0 [term 1 LEADER]: Becoming Leader. State: Replica: 1fa69357231b45588341634941318dc0, State: Running, Role: LEADER
I20260926 06:19:13.570751 23203 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1fa69357231b45588341634941318dc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fa69357231b45588341634941318dc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33637 } }
I20260926 06:19:13.571475 23204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fa69357231b45588341634941318dc0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fa69357231b45588341634941318dc0. Latest consensus state: current_term: 1 leader_uuid: "1fa69357231b45588341634941318dc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fa69357231b45588341634941318dc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33637 } } }
I20260926 06:19:13.571605 23204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fa69357231b45588341634941318dc0 [sys.catalog]: This master's current role is: LEADER
I20260926 06:19:13.571401 23203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fa69357231b45588341634941318dc0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1fa69357231b45588341634941318dc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fa69357231b45588341634941318dc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33637 } } }
I20260926 06:19:13.571816 23203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fa69357231b45588341634941318dc0 [sys.catalog]: This master's current role is: LEADER
W20260926 06:19:13.573433 23217 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1fa69357231b45588341634941318dc0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 06:19:13.574563 23217 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 06:19:13.574637 23202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:13.567111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52403 (local address 127.0.0.1:33637)
0926 06:19:13.567294 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:13.567302 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:13.567328 (+ 26us) server_negotiation.cc:407] Connection header received
0926 06:19:13.567932 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:13.567940 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:13.567996 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:13.568218 (+ 222us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:13.568539 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:13.570323 (+ 1784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:13.571499 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:13.572377 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:13.572477 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:13.573675 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:13.573684 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:13.573685 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:19:13.573913 (+ 228us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:13.573951 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:19:13.573967 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:13.574032 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:13.574205 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 06:19:13.574331 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":86,"threads_started":1}
I20260926 06:19:13.574647 23218 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:19:13.574872 23218 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:19:13.575778 2491 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:19:13.576928 23218 catalog_manager.cc:1347] Generated new cluster ID: 29f837e80bba411b8b445b2cfb7531d7
I20260926 06:19:13.576990 23218 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:19:13.585681 23202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:13.577682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49377 (local address 127.0.0.1:33637)
0926 06:19:13.577721 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:13.577723 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:13.581578 (+ 3855us) server_negotiation.cc:407] Connection header received
0926 06:19:13.581635 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:13.581636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:13.581649 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:13.581719 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:13.581901 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:13.582719 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:13.584166 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:13.584491 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:13.584570 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:13.585405 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:13.585406 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:13.585407 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:19:13.585412 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:13.585430 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:19:13.585435 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:13.585490 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:13.585558 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 06:19:13.585585 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 06:19:13.590019 23218 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:19:13.590816 23218 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:19:13.604058 23218 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1fa69357231b45588341634941318dc0: Generated new TSK 0
I20260926 06:19:13.604293 23218 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:19:13.611685 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 23130
2026-09-26T06:19:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1380 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790403292369990-2491-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:19:13 dist-test-slave-hndd krb5kdc[23229](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:19:13 dist-test-slave-hndd krb5kdc[23229](info): set up 2 sockets
Sep 26 06:19:13 dist-test-slave-hndd krb5kdc[23229](info): commencing operation
krb5kdc: starting...
W20260926 06:19:14.677569 2491 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Sep 26 06:19:14 dist-test-slave-hndd krb5kdc[23229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:19:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:19:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790403292369990-2491-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790403292369990-2491-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 06:19:14.809968 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37131
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790403292369990-2491-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37131
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:19:14.893637 23242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:19:14.893874 23242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:19:14.893901 23242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:19:14.895174 23242 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:19:14.895236 23242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:19:14.895251 23242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:19:14.895264 23242 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:19:14.895277 23242 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:19:14.896871 23242 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790403292369990-2491-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790403292369990-2491-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:19:14.897149 23242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:19:14.897465 23242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:19:14.902688 23250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:14.902706 23247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:14.902796 23248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:14.903131 23242 server_base.cc:1034] running on GCE node
I20260926 06:19:14.903379 23242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:19:14.903643 23242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:19:14.904799 23242 hybrid_clock.cc:648] HybridClock initialized: now 1790403554904750 us; error 70 us; skew 500 ppm
Sep 26 06:19:14 dist-test-slave-hndd krb5kdc[23229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:19:14.907773 23242 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 06:19:14.908334 23242 webserver.cc:460] Webserver started at http://127.0.0.1:36019/ using document root <none> and password file <none>
I20260926 06:19:14.908573 23242 fs_manager.cc:359] Metadata directory not provided
I20260926 06:19:14.908622 23242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:19:14.908800 23242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:19:14.909964 23242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790403292369990-2491-0/minicluster-data/master-0/data/instance:
uuid: "1b74cbbdcbb94377bd7d77fafea41297"
format_stamp: "Formatted at 2026-09-26 06:19:14 on dist-test-slave-hndd"
server_key: "ee3a7d790650c436938c4389785a00ea"
server_key_iv: "e51d9803af8c93d68e45bd632adb9561"
server_key_version: "encryptionkey@0"
I20260926 06:19:14.910336 23242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790403292369990-2491-0/minicluster-data/master-0/wal/instance:
uuid: "1b74cbbdcbb94377bd7d77fafea41297"
format_stamp: "Formatted at 2026-09-26 06:19:14 on dist-test-slave-hndd"
server_key: "ee3a7d790650c436938c4389785a00ea"
server_key_iv: "e51d9803af8c93d68e45bd632adb9561"
server_key_version: "encryptionkey@0"
I20260926 06:19:14.911870 23242 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:19:14.912756 23257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:14.912981 23242 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:19:14.913089 23242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "1b74cbbdcbb94377bd7d77fafea41297"
format_stamp: "Formatted at 2026-09-26 06:19:14 on dist-test-slave-hndd"
server_key: "ee3a7d790650c436938c4389785a00ea"
server_key_iv: "e51d9803af8c93d68e45bd632adb9561"
server_key_version: "encryptionkey@0"
I20260926 06:19:14.913183 23242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:19:14.916641 23242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:19:14.919108 23242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:19:14.919262 23242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:19:14.923404 23242 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37131
I20260926 06:19:14.923436 23309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37131 every 8 connection(s)
I20260926 06:19:14.923828 23242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:19:14.924817 23310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:19:14.925403 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 23242
I20260926 06:19:14.925518 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790403292369990-2491-0/minicluster-data/master-0/wal/instance
I20260926 06:19:14.925719 2491 external_mini_cluster.cc:1428] Setting key c41057532c7aee1cb9a669a352702ac0
I20260926 06:19:14.928085 23310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b74cbbdcbb94377bd7d77fafea41297: Bootstrap starting.
I20260926 06:19:14.928269 23313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:14.926264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35525 (local address 127.0.0.1:37131)
0926 06:19:14.926455 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:14.926462 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:14.926486 (+ 24us) server_negotiation.cc:407] Connection header received
0926 06:19:14.926900 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:14.926907 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:14.926931 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:14.927050 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:14.927713 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:14.927716 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:14.927716 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:19:14.927719 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:14.927760 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:19:14.927770 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:14.927832 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:14.927927 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 06:19:14.927979 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":89,"threads_started":1}
I20260926 06:19:14.928786 23310 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b74cbbdcbb94377bd7d77fafea41297: Neither blocks nor log segments found. Creating new log.
I20260926 06:19:14.929153 23310 log.cc:824] T 00000000000000000000000000000000 P 1b74cbbdcbb94377bd7d77fafea41297: Log is configured to *not* fsync() on all Append() calls
I20260926 06:19:14.929260 2491 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:19:14.929965 23310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b74cbbdcbb94377bd7d77fafea41297: No bootstrap required, opened a new log
I20260926 06:19:14.931860 23313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:14.930868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45401 (local address 127.0.0.1:37131)
0926 06:19:14.930905 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:14.930907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:14.930973 (+ 66us) server_negotiation.cc:407] Connection header received
0926 06:19:14.931036 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:14.931036 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:14.931040 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:14.931085 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:14.931605 (+ 520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:14.931606 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:14.931606 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:19:14.931609 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:14.931622 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:19:14.931625 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:14.931664 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:14.931729 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 06:19:14.931740 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 06:19:14.932619 23310 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b74cbbdcbb94377bd7d77fafea41297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b74cbbdcbb94377bd7d77fafea41297" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37131 } }
I20260926 06:19:14.932761 23310 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b74cbbdcbb94377bd7d77fafea41297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:19:14.932783 23310 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b74cbbdcbb94377bd7d77fafea41297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b74cbbdcbb94377bd7d77fafea41297, State: Initialized, Role: FOLLOWER
I20260926 06:19:14.932940 23310 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b74cbbdcbb94377bd7d77fafea41297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b74cbbdcbb94377bd7d77fafea41297" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37131 } }
I20260926 06:19:14.933028 23310 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b74cbbdcbb94377bd7d77fafea41297 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:19:14.933075 23310 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b74cbbdcbb94377bd7d77fafea41297 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:19:14.933126 23310 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b74cbbdcbb94377bd7d77fafea41297 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:19:14.933785 23310 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b74cbbdcbb94377bd7d77fafea41297 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b74cbbdcbb94377bd7d77fafea41297" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37131 } }
I20260926 06:19:14.933909 23310 leader_election.cc:302] T 00000000000000000000000000000000 P 1b74cbbdcbb94377bd7d77fafea41297 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b74cbbdcbb94377bd7d77fafea41297; no voters:
I20260926 06:19:14.934125 23310 leader_election.cc:288] T 00000000000000000000000000000000 P 1b74cbbdcbb94377bd7d77fafea41297 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:19:14.934412 23310 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b74cbbdcbb94377bd7d77fafea41297 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:19:14.934367 23320 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b74cbbdcbb94377bd7d77fafea41297 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:19:14.934845 23320 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b74cbbdcbb94377bd7d77fafea41297 [term 1 LEADER]: Becoming Leader. State: Replica: 1b74cbbdcbb94377bd7d77fafea41297, State: Running, Role: LEADER
I20260926 06:19:14.935153 23320 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b74cbbdcbb94377bd7d77fafea41297 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b74cbbdcbb94377bd7d77fafea41297" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37131 } }
I20260926 06:19:14.935631 23322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b74cbbdcbb94377bd7d77fafea41297 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b74cbbdcbb94377bd7d77fafea41297. Latest consensus state: current_term: 1 leader_uuid: "1b74cbbdcbb94377bd7d77fafea41297" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b74cbbdcbb94377bd7d77fafea41297" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37131 } } }
I20260926 06:19:14.935738 23322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b74cbbdcbb94377bd7d77fafea41297 [sys.catalog]: This master's current role is: LEADER
I20260926 06:19:14.935793 23321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b74cbbdcbb94377bd7d77fafea41297 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b74cbbdcbb94377bd7d77fafea41297" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b74cbbdcbb94377bd7d77fafea41297" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37131 } } }
I20260926 06:19:14.935854 23321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b74cbbdcbb94377bd7d77fafea41297 [sys.catalog]: This master's current role is: LEADER
I20260926 06:19:14.935942 23334 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:19:14.936661 23334 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:19:14.938234 23334 catalog_manager.cc:1347] Generated new cluster ID: b61d72fed6e3483da234867d1079d01a
I20260926 06:19:14.938293 23334 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:19:14.964269 23334 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:19:14.964466 23334 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:19:14.966789 23334 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b74cbbdcbb94377bd7d77fafea41297: Generated new TSK 0
I20260926 06:19:14.966998 23334 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:19:14.975775 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 23242
2026-09-26T06:19:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1363 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790403292369990-2491-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:19:15 dist-test-slave-hndd krb5kdc[23341](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:19:15 dist-test-slave-hndd krb5kdc[23341](info): set up 2 sockets
Sep 26 06:19:15 dist-test-slave-hndd krb5kdc[23341](info): commencing operation
krb5kdc: starting...
W20260926 06:19:16.021492 2491 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Sep 26 06:19:16 dist-test-slave-hndd krb5kdc[23341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:19:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:19:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790403292369990-2491-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790403292369990-2491-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 06:19:16.146503 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35951
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790403292369990-2491-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35951
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:19:16.227097 23354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:19:16.227380 23354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:19:16.227421 23354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:19:16.228605 23354 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:19:16.228682 23354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:19:16.228706 23354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:19:16.228729 23354 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:19:16.228760 23354 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:19:16.230316 23354 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790403292369990-2491-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35951
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790403292369990-2491-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:19:16.230615 23354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:19:16.230857 23354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:19:16.236304 23360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:16.236330 23362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:16.236349 23354 server_base.cc:1034] running on GCE node
W20260926 06:19:16.236351 23359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:16.236831 23354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:19:16.237154 23354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:19:16.238343 23354 hybrid_clock.cc:648] HybridClock initialized: now 1790403556238328 us; error 39 us; skew 500 ppm
Sep 26 06:19:16 dist-test-slave-hndd krb5kdc[23341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:19:16.241053 23354 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 06:19:16.241612 23354 webserver.cc:460] Webserver started at http://127.0.0.1:36111/ using document root <none> and password file <none>
I20260926 06:19:16.241873 23354 fs_manager.cc:359] Metadata directory not provided
I20260926 06:19:16.241947 23354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:19:16.242100 23354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:19:16.243192 23354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790403292369990-2491-0/minicluster-data/master-0/data/instance:
uuid: "9038f1f52208461c9b1df217ea4d95df"
format_stamp: "Formatted at 2026-09-26 06:19:16 on dist-test-slave-hndd"
server_key: "0e13f4a94f0f62cff0f163ecc79c92a6"
server_key_iv: "fd3cecaceece8177063f7f8b39505d50"
server_key_version: "encryptionkey@0"
I20260926 06:19:16.243558 23354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790403292369990-2491-0/minicluster-data/master-0/wal/instance:
uuid: "9038f1f52208461c9b1df217ea4d95df"
format_stamp: "Formatted at 2026-09-26 06:19:16 on dist-test-slave-hndd"
server_key: "0e13f4a94f0f62cff0f163ecc79c92a6"
server_key_iv: "fd3cecaceece8177063f7f8b39505d50"
server_key_version: "encryptionkey@0"
I20260926 06:19:16.244946 23354 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:19:16.245807 23369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:16.245993 23354 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 06:19:16.246102 23354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "9038f1f52208461c9b1df217ea4d95df"
format_stamp: "Formatted at 2026-09-26 06:19:16 on dist-test-slave-hndd"
server_key: "0e13f4a94f0f62cff0f163ecc79c92a6"
server_key_iv: "fd3cecaceece8177063f7f8b39505d50"
server_key_version: "encryptionkey@0"
I20260926 06:19:16.246182 23354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:19:16.255510 23354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:19:16.257925 23354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:19:16.258090 23354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:19:16.262523 23354 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35951
I20260926 06:19:16.262555 23421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35951 every 8 connection(s)
I20260926 06:19:16.262940 23354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:19:16.263978 23422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:19:16.266362 23422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9038f1f52208461c9b1df217ea4d95df: Bootstrap starting.
I20260926 06:19:16.266893 23422 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9038f1f52208461c9b1df217ea4d95df: Neither blocks nor log segments found. Creating new log.
I20260926 06:19:16.267154 23422 log.cc:824] T 00000000000000000000000000000000 P 9038f1f52208461c9b1df217ea4d95df: Log is configured to *not* fsync() on all Append() calls
I20260926 06:19:16.267866 23422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9038f1f52208461c9b1df217ea4d95df: No bootstrap required, opened a new log
I20260926 06:19:16.269532 23422 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9038f1f52208461c9b1df217ea4d95df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9038f1f52208461c9b1df217ea4d95df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35951 } }
I20260926 06:19:16.269670 23422 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9038f1f52208461c9b1df217ea4d95df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:19:16.269692 23422 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9038f1f52208461c9b1df217ea4d95df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9038f1f52208461c9b1df217ea4d95df, State: Initialized, Role: FOLLOWER
I20260926 06:19:16.269798 23422 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9038f1f52208461c9b1df217ea4d95df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9038f1f52208461c9b1df217ea4d95df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35951 } }
I20260926 06:19:16.269851 23422 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9038f1f52208461c9b1df217ea4d95df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:19:16.269872 23422 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9038f1f52208461c9b1df217ea4d95df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:19:16.269943 23422 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9038f1f52208461c9b1df217ea4d95df [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:19:16.270498 23422 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9038f1f52208461c9b1df217ea4d95df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9038f1f52208461c9b1df217ea4d95df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35951 } }
I20260926 06:19:16.270582 23422 leader_election.cc:302] T 00000000000000000000000000000000 P 9038f1f52208461c9b1df217ea4d95df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9038f1f52208461c9b1df217ea4d95df; no voters:
I20260926 06:19:16.270793 23422 leader_election.cc:288] T 00000000000000000000000000000000 P 9038f1f52208461c9b1df217ea4d95df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:19:16.270913 23425 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9038f1f52208461c9b1df217ea4d95df [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:19:16.271080 23422 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9038f1f52208461c9b1df217ea4d95df [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:19:16.271133 23425 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9038f1f52208461c9b1df217ea4d95df [term 1 LEADER]: Becoming Leader. State: Replica: 9038f1f52208461c9b1df217ea4d95df, State: Running, Role: LEADER
I20260926 06:19:16.271319 23425 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9038f1f52208461c9b1df217ea4d95df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9038f1f52208461c9b1df217ea4d95df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35951 } }
I20260926 06:19:16.271741 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 23354
I20260926 06:19:16.271865 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790403292369990-2491-0/minicluster-data/master-0/wal/instance
I20260926 06:19:16.271965 23427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9038f1f52208461c9b1df217ea4d95df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9038f1f52208461c9b1df217ea4d95df. Latest consensus state: current_term: 1 leader_uuid: "9038f1f52208461c9b1df217ea4d95df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9038f1f52208461c9b1df217ea4d95df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35951 } } }
I20260926 06:19:16.272044 23427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9038f1f52208461c9b1df217ea4d95df [sys.catalog]: This master's current role is: LEADER
I20260926 06:19:16.272065 2491 external_mini_cluster.cc:1428] Setting key 2439de83652548e5dadb49c6edb6b88c
I20260926 06:19:16.272504 23434 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:19:16.272606 23426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9038f1f52208461c9b1df217ea4d95df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9038f1f52208461c9b1df217ea4d95df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9038f1f52208461c9b1df217ea4d95df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35951 } } }
I20260926 06:19:16.272665 23426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9038f1f52208461c9b1df217ea4d95df [sys.catalog]: This master's current role is: LEADER
I20260926 06:19:16.273047 23434 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:19:16.274689 23434 catalog_manager.cc:1347] Generated new cluster ID: 3248c30f96544db1b13eb03bd0dadc0a
I20260926 06:19:16.274740 23434 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 06:19:16 dist-test-slave-hndd krb5kdc[23341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 06:19:16.284562 23442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:16.272625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:39579 (local address 127.0.0.1:35951)
0926 06:19:16.273017 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:16.273028 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:16.273061 (+ 33us) server_negotiation.cc:407] Connection header received
0926 06:19:16.273212 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:16.273222 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:16.273275 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:16.273463 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:16.274229 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:16.275605 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:16.276653 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:16.277288 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:16.277454 (+ 166us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:16.280341 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:16.280345 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:16.280345 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:19:16.280370 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:16.282595 (+ 2225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:16.283110 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:16.283112 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:16.283113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:16.283236 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:16.283529 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:16.283530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:16.283530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:16.283888 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:16.284047 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:16.284207 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 06:19:16.284316 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":65,"threads_started":1}
I20260926 06:19:16.285674 2491 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:19:16.286500 23434 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:19:16.287207 23434 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:19:16.293255 23434 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9038f1f52208461c9b1df217ea4d95df: Generated new TSK 0
I20260926 06:19:16.293498 23434 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:19:16.295616 23442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:16.287196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:49893 (local address 127.0.0.1:35951)
0926 06:19:16.287235 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:16.287236 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:16.287362 (+ 126us) server_negotiation.cc:407] Connection header received
0926 06:19:16.287440 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:16.287441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:16.287451 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:16.287513 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:16.288146 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:16.288974 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:16.290060 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:16.290402 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:16.290470 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:16.292034 (+ 1564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:16.292037 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:16.292037 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:19:16.292050 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:16.294249 (+ 2199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:16.294691 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:16.294693 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:16.294693 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:16.294764 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:16.295034 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:16.295035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:16.295035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:16.295176 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:16.295303 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:16.295442 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 06:19:16.295512 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 06:19:16.297796 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 23354
W20260926 06:19:16.302078 23339 connection.cc:535] client connection to 127.0.0.1:35951 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35951): Connection reset by peer (error 104)
2026-09-26T06:19:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1321 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790403292369990-2491-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:19:16 dist-test-slave-hndd krb5kdc[23453](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:19:16 dist-test-slave-hndd krb5kdc[23453](info): set up 2 sockets
Sep 26 06:19:16 dist-test-slave-hndd krb5kdc[23453](info): commencing operation
krb5kdc: starting...
W20260926 06:19:17.349747 2491 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Sep 26 06:19:17 dist-test-slave-hndd krb5kdc[23453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:19:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:19:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790403292369990-2491-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790403292369990-2491-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 06:19:17.477259 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33677
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790403292369990-2491-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33677
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:19:17.563858 23466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:19:17.564127 23466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:19:17.564167 23466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:19:17.565702 23466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:19:17.565768 23466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:19:17.565793 23466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:19:17.565828 23466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:19:17.565856 23466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:19:17.567380 23466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790403292369990-2491-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33677
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790403292369990-2491-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:19:17.567677 23466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:19:17.567919 23466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:19:17.573104 23474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:17.573123 23466 server_base.cc:1034] running on GCE node
W20260926 06:19:17.573064 23471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:17.573091 23472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:17.573578 23466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:19:17.573901 23466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:19:17.575095 23466 hybrid_clock.cc:648] HybridClock initialized: now 1790403557575077 us; error 48 us; skew 500 ppm
Sep 26 06:19:17 dist-test-slave-hndd krb5kdc[23453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403557, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:19:17.577618 23466 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 06:19:17.578068 23466 webserver.cc:460] Webserver started at http://127.0.0.1:35407/ using document root <none> and password file <none>
I20260926 06:19:17.578308 23466 fs_manager.cc:359] Metadata directory not provided
I20260926 06:19:17.578377 23466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:19:17.578524 23466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:19:17.579595 23466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790403292369990-2491-0/minicluster-data/master-0/data/instance:
uuid: "9b0563c772b648f88e13c4557d367a4a"
format_stamp: "Formatted at 2026-09-26 06:19:17 on dist-test-slave-hndd"
server_key: "7076ac76240f3f7c3c94493c191d8b39"
server_key_iv: "994c32787dc58ca8f9d61e13c09b416f"
server_key_version: "encryptionkey@0"
I20260926 06:19:17.579952 23466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790403292369990-2491-0/minicluster-data/master-0/wal/instance:
uuid: "9b0563c772b648f88e13c4557d367a4a"
format_stamp: "Formatted at 2026-09-26 06:19:17 on dist-test-slave-hndd"
server_key: "7076ac76240f3f7c3c94493c191d8b39"
server_key_iv: "994c32787dc58ca8f9d61e13c09b416f"
server_key_version: "encryptionkey@0"
I20260926 06:19:17.581316 23466 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:19:17.582222 23481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:17.582479 23466 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:19:17.582558 23466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "9b0563c772b648f88e13c4557d367a4a"
format_stamp: "Formatted at 2026-09-26 06:19:17 on dist-test-slave-hndd"
server_key: "7076ac76240f3f7c3c94493c191d8b39"
server_key_iv: "994c32787dc58ca8f9d61e13c09b416f"
server_key_version: "encryptionkey@0"
I20260926 06:19:17.582612 23466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:19:17.602540 23466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:19:17.604573 23466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:19:17.604689 23466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:19:17.608875 23466 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33677
I20260926 06:19:17.608906 23533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33677 every 8 connection(s)
I20260926 06:19:17.609267 23466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:19:17.610265 23534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:19:17.612494 23534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b0563c772b648f88e13c4557d367a4a: Bootstrap starting.
I20260926 06:19:17.612921 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 23466
I20260926 06:19:17.613057 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790403292369990-2491-0/minicluster-data/master-0/wal/instance
I20260926 06:19:17.613152 23534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b0563c772b648f88e13c4557d367a4a: Neither blocks nor log segments found. Creating new log.
I20260926 06:19:17.613296 2491 external_mini_cluster.cc:1428] Setting key 5a5c865c0e25155616be63163337a113
I20260926 06:19:17.613556 23534 log.cc:824] T 00000000000000000000000000000000 P 9b0563c772b648f88e13c4557d367a4a: Log is configured to *not* fsync() on all Append() calls
I20260926 06:19:17.614562 23534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b0563c772b648f88e13c4557d367a4a: No bootstrap required, opened a new log
I20260926 06:19:17.616461 23534 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b0563c772b648f88e13c4557d367a4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b0563c772b648f88e13c4557d367a4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33677 } }
I20260926 06:19:17.616619 23534 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b0563c772b648f88e13c4557d367a4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:19:17.616671 23534 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b0563c772b648f88e13c4557d367a4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b0563c772b648f88e13c4557d367a4a, State: Initialized, Role: FOLLOWER
I20260926 06:19:17.616825 23534 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b0563c772b648f88e13c4557d367a4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b0563c772b648f88e13c4557d367a4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33677 } }
I20260926 06:19:17.616914 23534 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b0563c772b648f88e13c4557d367a4a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:19:17.616945 23534 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b0563c772b648f88e13c4557d367a4a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:19:17.616981 23534 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b0563c772b648f88e13c4557d367a4a [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:19:17.617756 23534 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b0563c772b648f88e13c4557d367a4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b0563c772b648f88e13c4557d367a4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33677 } }
I20260926 06:19:17.617880 23534 leader_election.cc:302] T 00000000000000000000000000000000 P 9b0563c772b648f88e13c4557d367a4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b0563c772b648f88e13c4557d367a4a; no voters:
I20260926 06:19:17.618083 23534 leader_election.cc:288] T 00000000000000000000000000000000 P 9b0563c772b648f88e13c4557d367a4a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:19:17.618256 23539 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b0563c772b648f88e13c4557d367a4a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:19:17.618373 23534 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b0563c772b648f88e13c4557d367a4a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:19:17.618425 23539 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b0563c772b648f88e13c4557d367a4a [term 1 LEADER]: Becoming Leader. State: Replica: 9b0563c772b648f88e13c4557d367a4a, State: Running, Role: LEADER
I20260926 06:19:17.618525 23539 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b0563c772b648f88e13c4557d367a4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b0563c772b648f88e13c4557d367a4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33677 } }
I20260926 06:19:17.618973 23540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b0563c772b648f88e13c4557d367a4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b0563c772b648f88e13c4557d367a4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b0563c772b648f88e13c4557d367a4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33677 } } }
I20260926 06:19:17.619064 23540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b0563c772b648f88e13c4557d367a4a [sys.catalog]: This master's current role is: LEADER
I20260926 06:19:17.619540 23541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b0563c772b648f88e13c4557d367a4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b0563c772b648f88e13c4557d367a4a. Latest consensus state: current_term: 1 leader_uuid: "9b0563c772b648f88e13c4557d367a4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b0563c772b648f88e13c4557d367a4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33677 } } }
I20260926 06:19:17.619648 23541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b0563c772b648f88e13c4557d367a4a [sys.catalog]: This master's current role is: LEADER
I20260926 06:19:17.619717 23538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:17.613969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:59833 (local address 127.0.0.1:33677)
0926 06:19:17.614146 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:17.614153 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:17.614182 (+ 29us) server_negotiation.cc:407] Connection header received
0926 06:19:17.614601 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:17.614606 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:17.614691 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:17.614866 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:17.615091 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:17.615968 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:17.616794 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:17.617453 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:17.617576 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:17.618426 (+ 850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:17.618430 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:17.618431 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:19:17.618650 (+ 219us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:17.618689 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:19:17.618704 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:17.618769 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:17.618964 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 06:19:17.618996 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":89,"threads_started":1}
I20260926 06:19:17.619505 23548 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:19:17.620229 23548 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:19:17.620932 2491 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:19:17.622656 23548 catalog_manager.cc:1347] Generated new cluster ID: 8a7bb3eda92e41e48449353cc69aa107
I20260926 06:19:17.622713 23548 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:19:17.626149 23538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:17.622445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:53601 (local address 127.0.0.1:33677)
0926 06:19:17.622480 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:17.622482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:17.622633 (+ 151us) server_negotiation.cc:407] Connection header received
0926 06:19:17.622742 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:17.622743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:17.622756 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:17.622787 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:17.623002 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:17.623721 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:17.624664 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:17.625005 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:17.625045 (+ 40us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:17.625912 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:17.625913 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:17.625913 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:19:17.625917 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:17.625930 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:19:17.625934 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:17.625962 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:17.626069 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 06:19:17.626080 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 06:19:17.633554 23548 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:19:17.634152 23548 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:19:17.636458 23548 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b0563c772b648f88e13c4557d367a4a: Generated new TSK 0
I20260926 06:19:17.636639 23548 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:19:17.638700 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 23466
W20260926 06:19:17.643260 23451 connection.cc:535] client connection to 127.0.0.1:33677 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33677): Connection reset by peer (error 104)
2026-09-26T06:19:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1341 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790403292369990-2491-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:19:17 dist-test-slave-hndd krb5kdc[23565](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:19:17 dist-test-slave-hndd krb5kdc[23565](info): set up 2 sockets
Sep 26 06:19:17 dist-test-slave-hndd krb5kdc[23565](info): commencing operation
krb5kdc: starting...
W20260926 06:19:18.693514 2491 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Sep 26 06:19:18 dist-test-slave-hndd krb5kdc[23565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:19:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:19:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790403292369990-2491-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790403292369990-2491-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 06:19:18.816723 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35341
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790403292369990-2491-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:19:18.901046 23578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:19:18.901332 23578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:19:18.901403 23578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:19:18.902685 23578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:19:18.902769 23578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:19:18.902797 23578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:19:18.902827 23578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:19:18.902861 23578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:19:18.904443 23578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790403292369990-2491-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790403292369990-2491-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:19:18.904740 23578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:19:18.904987 23578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:19:18.910096 23584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:18.910089 23586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:18.910089 23583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:18.910218 23578 server_base.cc:1034] running on GCE node
I20260926 06:19:18.910550 23578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:19:18.910877 23578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:19:18.912045 23578 hybrid_clock.cc:648] HybridClock initialized: now 1790403558912023 us; error 34 us; skew 500 ppm
Sep 26 06:19:18 dist-test-slave-hndd krb5kdc[23565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:19:18.917739 23578 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 06:19:18.918236 23578 webserver.cc:460] Webserver started at http://127.0.0.1:45345/ using document root <none> and password file <none>
I20260926 06:19:18.918503 23578 fs_manager.cc:359] Metadata directory not provided
I20260926 06:19:18.918576 23578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:19:18.918723 23578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:19:18.919785 23578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790403292369990-2491-0/minicluster-data/master-0/data/instance:
uuid: "a15d0aa55627474fb75bb85ed490190c"
format_stamp: "Formatted at 2026-09-26 06:19:18 on dist-test-slave-hndd"
server_key: "e2fa20c25491a6be51fc08a3ce2056e6"
server_key_iv: "9ef22369159243522aa409c5c3aa2595"
server_key_version: "encryptionkey@0"
I20260926 06:19:18.920140 23578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790403292369990-2491-0/minicluster-data/master-0/wal/instance:
uuid: "a15d0aa55627474fb75bb85ed490190c"
format_stamp: "Formatted at 2026-09-26 06:19:18 on dist-test-slave-hndd"
server_key: "e2fa20c25491a6be51fc08a3ce2056e6"
server_key_iv: "9ef22369159243522aa409c5c3aa2595"
server_key_version: "encryptionkey@0"
I20260926 06:19:18.921556 23578 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 06:19:18.922428 23593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:18.922653 23578 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:19:18.922765 23578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "a15d0aa55627474fb75bb85ed490190c"
format_stamp: "Formatted at 2026-09-26 06:19:18 on dist-test-slave-hndd"
server_key: "e2fa20c25491a6be51fc08a3ce2056e6"
server_key_iv: "9ef22369159243522aa409c5c3aa2595"
server_key_version: "encryptionkey@0"
I20260926 06:19:18.922849 23578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:19:18.926381 23578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:19:18.928651 23578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:19:18.928817 23578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:19:18.933154 23578 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35341
I20260926 06:19:18.933192 23645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35341 every 8 connection(s)
I20260926 06:19:18.933673 23578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:19:18.934718 23646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:19:18.936947 23646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a15d0aa55627474fb75bb85ed490190c: Bootstrap starting.
I20260926 06:19:18.937592 23646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a15d0aa55627474fb75bb85ed490190c: Neither blocks nor log segments found. Creating new log.
I20260926 06:19:18.937879 23646 log.cc:824] T 00000000000000000000000000000000 P a15d0aa55627474fb75bb85ed490190c: Log is configured to *not* fsync() on all Append() calls
I20260926 06:19:18.938561 23646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a15d0aa55627474fb75bb85ed490190c: No bootstrap required, opened a new log
I20260926 06:19:18.940193 23646 raft_consensus.cc:348] T 00000000000000000000000000000000 P a15d0aa55627474fb75bb85ed490190c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a15d0aa55627474fb75bb85ed490190c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35341 } }
I20260926 06:19:18.940353 23646 raft_consensus.cc:374] T 00000000000000000000000000000000 P a15d0aa55627474fb75bb85ed490190c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:19:18.940376 23646 raft_consensus.cc:729] T 00000000000000000000000000000000 P a15d0aa55627474fb75bb85ed490190c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a15d0aa55627474fb75bb85ed490190c, State: Initialized, Role: FOLLOWER
I20260926 06:19:18.940549 23646 consensus_queue.cc:260] T 00000000000000000000000000000000 P a15d0aa55627474fb75bb85ed490190c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a15d0aa55627474fb75bb85ed490190c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35341 } }
I20260926 06:19:18.940611 23646 raft_consensus.cc:388] T 00000000000000000000000000000000 P a15d0aa55627474fb75bb85ed490190c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:19:18.940634 23646 raft_consensus.cc:482] T 00000000000000000000000000000000 P a15d0aa55627474fb75bb85ed490190c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:19:18.940701 23646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a15d0aa55627474fb75bb85ed490190c [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:19:18.941237 23646 raft_consensus.cc:504] T 00000000000000000000000000000000 P a15d0aa55627474fb75bb85ed490190c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a15d0aa55627474fb75bb85ed490190c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35341 } }
I20260926 06:19:18.941354 23646 leader_election.cc:302] T 00000000000000000000000000000000 P a15d0aa55627474fb75bb85ed490190c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a15d0aa55627474fb75bb85ed490190c; no voters:
I20260926 06:19:18.941570 23646 leader_election.cc:288] T 00000000000000000000000000000000 P a15d0aa55627474fb75bb85ed490190c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:19:18.941638 23649 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a15d0aa55627474fb75bb85ed490190c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:19:18.941812 23649 raft_consensus.cc:686] T 00000000000000000000000000000000 P a15d0aa55627474fb75bb85ed490190c [term 1 LEADER]: Becoming Leader. State: Replica: a15d0aa55627474fb75bb85ed490190c, State: Running, Role: LEADER
I20260926 06:19:18.941836 23646 sys_catalog.cc:564] T 00000000000000000000000000000000 P a15d0aa55627474fb75bb85ed490190c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:19:18.941977 23649 consensus_queue.cc:237] T 00000000000000000000000000000000 P a15d0aa55627474fb75bb85ed490190c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a15d0aa55627474fb75bb85ed490190c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35341 } }
I20260926 06:19:18.942139 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 23578
I20260926 06:19:18.942292 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790403292369990-2491-0/minicluster-data/master-0/wal/instance
I20260926 06:19:18.942524 2491 external_mini_cluster.cc:1428] Setting key c8d00ae87ebb8c947bd62289e40a7ccc
I20260926 06:19:18.943182 23651 sys_catalog.cc:455] T 00000000000000000000000000000000 P a15d0aa55627474fb75bb85ed490190c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a15d0aa55627474fb75bb85ed490190c. Latest consensus state: current_term: 1 leader_uuid: "a15d0aa55627474fb75bb85ed490190c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a15d0aa55627474fb75bb85ed490190c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35341 } } }
I20260926 06:19:18.943295 23651 sys_catalog.cc:458] T 00000000000000000000000000000000 P a15d0aa55627474fb75bb85ed490190c [sys.catalog]: This master's current role is: LEADER
I20260926 06:19:18.943393 23650 sys_catalog.cc:455] T 00000000000000000000000000000000 P a15d0aa55627474fb75bb85ed490190c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a15d0aa55627474fb75bb85ed490190c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a15d0aa55627474fb75bb85ed490190c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35341 } } }
I20260926 06:19:18.943476 23650 sys_catalog.cc:458] T 00000000000000000000000000000000 P a15d0aa55627474fb75bb85ed490190c [sys.catalog]: This master's current role is: LEADER
I20260926 06:19:18.943599 23661 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:19:18.944162 23661 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:19:18.945225 23660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:18.943049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:60515 (local address 127.0.0.1:35341)
0926 06:19:18.943601 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:18.943611 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:18.943647 (+ 36us) server_negotiation.cc:407] Connection header received
0926 06:19:18.943774 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:18.943783 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:18.943825 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:18.943987 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:18.944765 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:18.944768 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:18.944769 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:19:18.944773 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:18.944832 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:19:18.944863 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:18.944931 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:18.945010 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 06:19:18.945052 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":89,"threads_started":1}
I20260926 06:19:18.945875 23661 catalog_manager.cc:1347] Generated new cluster ID: 92a6306fef3c478a8f9ccef15188d7e9
I20260926 06:19:18.945926 23661 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:19:18.946188 2491 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:19:18.948856 23660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:18.947631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:40135 (local address 127.0.0.1:35341)
0926 06:19:18.947662 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:18.947663 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:18.947874 (+ 211us) server_negotiation.cc:407] Connection header received
0926 06:19:18.947954 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:18.947955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:18.947959 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:18.947987 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:18.948651 (+ 664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:18.948652 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:18.948653 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:19:18.948656 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:18.948669 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:19:18.948673 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:18.948698 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:18.948783 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 06:19:18.948795 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260926 06:19:18.964458 23661 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:19:18.964701 23661 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:19:18.966898 23661 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a15d0aa55627474fb75bb85ed490190c: Generated new TSK 0
I20260926 06:19:18.967108 23661 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:19:18.968964 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 23578
2026-09-26T06:19:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1330 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790403292369990-2491-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:19:19 dist-test-slave-hndd krb5kdc[23677](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:19:19 dist-test-slave-hndd krb5kdc[23677](info): set up 2 sockets
Sep 26 06:19:19 dist-test-slave-hndd krb5kdc[23677](info): commencing operation
krb5kdc: starting...
W20260926 06:19:20.037467 2491 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Sep 26 06:19:20 dist-test-slave-hndd krb5kdc[23677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:19:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:19:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790403292369990-2491-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790403292369990-2491-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 06:19:20.166571 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38799
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790403292369990-2491-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38799
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:19:20.250192 23690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:19:20.250438 23690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:19:20.250473 23690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:19:20.251786 23690 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:19:20.251842 23690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:19:20.251857 23690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:19:20.251870 23690 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:19:20.251883 23690 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:19:20.253453 23690 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790403292369990-2491-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38799
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790403292369990-2491-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:19:20.253671 23690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:19:20.253865 23690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:19:20.259099 23696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:20.259145 23698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:20.259088 23695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:20.259347 23690 server_base.cc:1034] running on GCE node
I20260926 06:19:20.259635 23690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:19:20.259972 23690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:19:20.261219 23690 hybrid_clock.cc:648] HybridClock initialized: now 1790403560261181 us; error 39 us; skew 500 ppm
Sep 26 06:19:20 dist-test-slave-hndd krb5kdc[23677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:19:20.264845 23690 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 06:19:20.265466 23690 webserver.cc:460] Webserver started at http://127.0.0.1:36991/ using document root <none> and password file <none>
I20260926 06:19:20.265695 23690 fs_manager.cc:359] Metadata directory not provided
I20260926 06:19:20.265749 23690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:19:20.265846 23690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:19:20.266909 23690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790403292369990-2491-0/minicluster-data/master-0/data/instance:
uuid: "0e9bf6e5bec143dc91931b15432a0744"
format_stamp: "Formatted at 2026-09-26 06:19:20 on dist-test-slave-hndd"
server_key: "184da140eae3e037a80bac8a75236c30"
server_key_iv: "0860f2863adf09aff4014f56623b5502"
server_key_version: "encryptionkey@0"
I20260926 06:19:20.267225 23690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790403292369990-2491-0/minicluster-data/master-0/wal/instance:
uuid: "0e9bf6e5bec143dc91931b15432a0744"
format_stamp: "Formatted at 2026-09-26 06:19:20 on dist-test-slave-hndd"
server_key: "184da140eae3e037a80bac8a75236c30"
server_key_iv: "0860f2863adf09aff4014f56623b5502"
server_key_version: "encryptionkey@0"
I20260926 06:19:20.268544 23690 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:19:20.269496 23705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:20.269714 23690 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:19:20.269833 23690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "0e9bf6e5bec143dc91931b15432a0744"
format_stamp: "Formatted at 2026-09-26 06:19:20 on dist-test-slave-hndd"
server_key: "184da140eae3e037a80bac8a75236c30"
server_key_iv: "0860f2863adf09aff4014f56623b5502"
server_key_version: "encryptionkey@0"
I20260926 06:19:20.269928 23690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:19:20.295977 23690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:19:20.298571 23690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:19:20.298707 23690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:19:20.302887 23690 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38799
I20260926 06:19:20.303107 23757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38799 every 8 connection(s)
I20260926 06:19:20.303366 23690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:19:20.304283 23758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:19:20.306632 23758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e9bf6e5bec143dc91931b15432a0744: Bootstrap starting.
I20260926 06:19:20.307243 23758 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e9bf6e5bec143dc91931b15432a0744: Neither blocks nor log segments found. Creating new log.
I20260926 06:19:20.307552 23758 log.cc:824] T 00000000000000000000000000000000 P 0e9bf6e5bec143dc91931b15432a0744: Log is configured to *not* fsync() on all Append() calls
I20260926 06:19:20.308284 23758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e9bf6e5bec143dc91931b15432a0744: No bootstrap required, opened a new log
I20260926 06:19:20.309926 23758 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e9bf6e5bec143dc91931b15432a0744 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e9bf6e5bec143dc91931b15432a0744" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38799 } }
I20260926 06:19:20.310070 23758 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e9bf6e5bec143dc91931b15432a0744 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:19:20.310093 23758 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e9bf6e5bec143dc91931b15432a0744 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e9bf6e5bec143dc91931b15432a0744, State: Initialized, Role: FOLLOWER
I20260926 06:19:20.310266 23758 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e9bf6e5bec143dc91931b15432a0744 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e9bf6e5bec143dc91931b15432a0744" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38799 } }
I20260926 06:19:20.310333 23758 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e9bf6e5bec143dc91931b15432a0744 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:19:20.310355 23758 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e9bf6e5bec143dc91931b15432a0744 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:19:20.310405 23758 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e9bf6e5bec143dc91931b15432a0744 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:19:20.310956 23758 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e9bf6e5bec143dc91931b15432a0744 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e9bf6e5bec143dc91931b15432a0744" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38799 } }
I20260926 06:19:20.311074 23758 leader_election.cc:302] T 00000000000000000000000000000000 P 0e9bf6e5bec143dc91931b15432a0744 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e9bf6e5bec143dc91931b15432a0744; no voters:
I20260926 06:19:20.311269 23758 leader_election.cc:288] T 00000000000000000000000000000000 P 0e9bf6e5bec143dc91931b15432a0744 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:19:20.311311 23761 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e9bf6e5bec143dc91931b15432a0744 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:19:20.311518 23761 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e9bf6e5bec143dc91931b15432a0744 [term 1 LEADER]: Becoming Leader. State: Replica: 0e9bf6e5bec143dc91931b15432a0744, State: Running, Role: LEADER
I20260926 06:19:20.311547 23758 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e9bf6e5bec143dc91931b15432a0744 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:19:20.311621 23761 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e9bf6e5bec143dc91931b15432a0744 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e9bf6e5bec143dc91931b15432a0744" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38799 } }
I20260926 06:19:20.312009 23762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e9bf6e5bec143dc91931b15432a0744 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e9bf6e5bec143dc91931b15432a0744" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e9bf6e5bec143dc91931b15432a0744" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38799 } } }
I20260926 06:19:20.312119 23762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e9bf6e5bec143dc91931b15432a0744 [sys.catalog]: This master's current role is: LEADER
I20260926 06:19:20.312011 23763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e9bf6e5bec143dc91931b15432a0744 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e9bf6e5bec143dc91931b15432a0744. Latest consensus state: current_term: 1 leader_uuid: "0e9bf6e5bec143dc91931b15432a0744" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e9bf6e5bec143dc91931b15432a0744" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38799 } } }
I20260926 06:19:20.312251 23763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e9bf6e5bec143dc91931b15432a0744 [sys.catalog]: This master's current role is: LEADER
I20260926 06:19:20.312347 23765 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:19:20.312577 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 23690
I20260926 06:19:20.312716 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790403292369990-2491-0/minicluster-data/master-0/wal/instance
I20260926 06:19:20.312928 2491 external_mini_cluster.cc:1428] Setting key 32678b6ac0c9ca1d822186a05f09461a
I20260926 06:19:20.312951 23765 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:19:20.314878 23765 catalog_manager.cc:1347] Generated new cluster ID: 0c9dcd806a0049188a9e21df649796f6
I20260926 06:19:20.314934 23765 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 06:19:20 dist-test-slave-hndd krb5kdc[23677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 06:19:20.325591 23778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:20.313510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:44161 (local address 127.0.0.1:38799)
0926 06:19:20.313874 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:20.313883 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:20.313984 (+ 101us) server_negotiation.cc:407] Connection header received
0926 06:19:20.314104 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:20.314111 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:20.314171 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:20.314499 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:20.315134 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:20.316609 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:20.317679 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:20.318326 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:20.318445 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:20.321506 (+ 3061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:20.321512 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:20.321512 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:19:20.321527 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:20.323613 (+ 2086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:20.324114 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:20.324117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:20.324118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:20.324249 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:20.324507 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:20.324507 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:20.324508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:20.324817 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:20.324979 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:20.325108 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 06:19:20.325315 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":76,"threads_started":1}
I20260926 06:19:20.326722 2491 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:19:20.327983 23765 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:19:20.328697 23765 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:19:20.334039 23765 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e9bf6e5bec143dc91931b15432a0744: Generated new TSK 0
I20260926 06:19:20.334357 23765 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:19:20.335816 23778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:20.328206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:32857 (local address 127.0.0.1:38799)
0926 06:19:20.328248 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:20.328250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:20.328390 (+ 140us) server_negotiation.cc:407] Connection header received
0926 06:19:20.328421 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:20.328422 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:20.328432 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:20.328489 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:20.329154 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:20.329919 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:20.330666 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:20.330899 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:20.330980 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:20.332601 (+ 1621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:20.332602 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:20.332603 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:19:20.332612 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:20.334569 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:20.334988 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:20.334989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:20.334989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:20.335053 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:20.335317 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:20.335318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:20.335319 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:20.335453 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:20.335571 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:20.335649 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 06:19:20.335714 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 06:19:20.338264 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 23690
W20260926 06:19:20.342908 23675 connection.cc:535] client connection to 127.0.0.1:38799 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38799): Connection reset by peer (error 104)
2026-09-26T06:19:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1369 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790403292369990-2491-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:19:20 dist-test-slave-hndd krb5kdc[23789](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:19:20 dist-test-slave-hndd krb5kdc[23789](info): set up 2 sockets
Sep 26 06:19:20 dist-test-slave-hndd krb5kdc[23789](info): commencing operation
krb5kdc: starting...
W20260926 06:19:21.425653 2491 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Sep 26 06:19:21 dist-test-slave-hndd krb5kdc[23789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:19:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:19:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790403292369990-2491-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790403292369990-2491-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 06:19:21.556823 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45373
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790403292369990-2491-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45373
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:19:21.648113 23802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:19:21.648335 23802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:19:21.648360 23802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:19:21.649659 23802 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:19:21.649708 23802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:19:21.649724 23802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:19:21.649739 23802 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:19:21.649752 23802 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:19:21.651391 23802 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790403292369990-2491-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45373
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790403292369990-2491-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:19:21.651659 23802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:19:21.651917 23802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:19:21.657265 23810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:21.657513 23802 server_base.cc:1034] running on GCE node
W20260926 06:19:21.657317 23807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:21.657361 23808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:21.658361 23802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:19:21.658852 23802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:19:21.660117 23802 hybrid_clock.cc:648] HybridClock initialized: now 1790403561660089 us; error 78 us; skew 500 ppm
Sep 26 06:19:21 dist-test-slave-hndd krb5kdc[23789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:19:21.663072 23802 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 06:19:21.663628 23802 webserver.cc:460] Webserver started at http://127.0.0.1:45871/ using document root <none> and password file <none>
I20260926 06:19:21.663925 23802 fs_manager.cc:359] Metadata directory not provided
I20260926 06:19:21.663997 23802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:19:21.664156 23802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:19:21.665256 23802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790403292369990-2491-0/minicluster-data/master-0/data/instance:
uuid: "40b782f443a34b979db4cf7a836c90a3"
format_stamp: "Formatted at 2026-09-26 06:19:21 on dist-test-slave-hndd"
server_key: "044f0378cf5dcf455f43e003f82671b5"
server_key_iv: "ee8d3eeac1b3b504aaa03bfec4c4aff5"
server_key_version: "encryptionkey@0"
I20260926 06:19:21.665691 23802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790403292369990-2491-0/minicluster-data/master-0/wal/instance:
uuid: "40b782f443a34b979db4cf7a836c90a3"
format_stamp: "Formatted at 2026-09-26 06:19:21 on dist-test-slave-hndd"
server_key: "044f0378cf5dcf455f43e003f82671b5"
server_key_iv: "ee8d3eeac1b3b504aaa03bfec4c4aff5"
server_key_version: "encryptionkey@0"
I20260926 06:19:21.667320 23802 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:19:21.668354 23817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:21.668642 23802 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:19:21.668784 23802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "40b782f443a34b979db4cf7a836c90a3"
format_stamp: "Formatted at 2026-09-26 06:19:21 on dist-test-slave-hndd"
server_key: "044f0378cf5dcf455f43e003f82671b5"
server_key_iv: "ee8d3eeac1b3b504aaa03bfec4c4aff5"
server_key_version: "encryptionkey@0"
I20260926 06:19:21.668888 23802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:19:21.676895 23802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:19:21.679232 23802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:19:21.679430 23802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:19:21.684028 23802 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45373
I20260926 06:19:21.684073 23869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45373 every 8 connection(s)
I20260926 06:19:21.684451 23802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:19:21.685299 23870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:19:21.687623 23870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40b782f443a34b979db4cf7a836c90a3: Bootstrap starting.
I20260926 06:19:21.688160 23870 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40b782f443a34b979db4cf7a836c90a3: Neither blocks nor log segments found. Creating new log.
I20260926 06:19:21.688400 23870 log.cc:824] T 00000000000000000000000000000000 P 40b782f443a34b979db4cf7a836c90a3: Log is configured to *not* fsync() on all Append() calls
I20260926 06:19:21.689047 23870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40b782f443a34b979db4cf7a836c90a3: No bootstrap required, opened a new log
I20260926 06:19:21.690704 23870 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40b782f443a34b979db4cf7a836c90a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b782f443a34b979db4cf7a836c90a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45373 } }
I20260926 06:19:21.690865 23870 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40b782f443a34b979db4cf7a836c90a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:19:21.690888 23870 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40b782f443a34b979db4cf7a836c90a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40b782f443a34b979db4cf7a836c90a3, State: Initialized, Role: FOLLOWER
I20260926 06:19:21.690991 23870 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40b782f443a34b979db4cf7a836c90a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b782f443a34b979db4cf7a836c90a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45373 } }
I20260926 06:19:21.691043 23870 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40b782f443a34b979db4cf7a836c90a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:19:21.691064 23870 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40b782f443a34b979db4cf7a836c90a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:19:21.691138 23870 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40b782f443a34b979db4cf7a836c90a3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:19:21.691680 23870 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40b782f443a34b979db4cf7a836c90a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b782f443a34b979db4cf7a836c90a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45373 } }
I20260926 06:19:21.691766 23870 leader_election.cc:302] T 00000000000000000000000000000000 P 40b782f443a34b979db4cf7a836c90a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40b782f443a34b979db4cf7a836c90a3; no voters:
I20260926 06:19:21.691936 23870 leader_election.cc:288] T 00000000000000000000000000000000 P 40b782f443a34b979db4cf7a836c90a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:19:21.692087 23873 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40b782f443a34b979db4cf7a836c90a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:19:21.692230 23870 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40b782f443a34b979db4cf7a836c90a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:19:21.692284 23873 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40b782f443a34b979db4cf7a836c90a3 [term 1 LEADER]: Becoming Leader. State: Replica: 40b782f443a34b979db4cf7a836c90a3, State: Running, Role: LEADER
I20260926 06:19:21.692390 23873 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40b782f443a34b979db4cf7a836c90a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b782f443a34b979db4cf7a836c90a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45373 } }
I20260926 06:19:21.692485 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 23802
I20260926 06:19:21.692612 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790403292369990-2491-0/minicluster-data/master-0/wal/instance
I20260926 06:19:21.692760 23873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40b782f443a34b979db4cf7a836c90a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40b782f443a34b979db4cf7a836c90a3. Latest consensus state: current_term: 1 leader_uuid: "40b782f443a34b979db4cf7a836c90a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b782f443a34b979db4cf7a836c90a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45373 } } }
I20260926 06:19:21.692845 23873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40b782f443a34b979db4cf7a836c90a3 [sys.catalog]: This master's current role is: LEADER
I20260926 06:19:21.692859 2491 external_mini_cluster.cc:1428] Setting key 2e652952e577e56f7569ca29d20c5b9f
I20260926 06:19:21.692943 23874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40b782f443a34b979db4cf7a836c90a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40b782f443a34b979db4cf7a836c90a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b782f443a34b979db4cf7a836c90a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45373 } } }
I20260926 06:19:21.693005 23874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40b782f443a34b979db4cf7a836c90a3 [sys.catalog]: This master's current role is: LEADER
I20260926 06:19:21.693251 23881 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:19:21.693867 23881 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:19:21.696151 23881 catalog_manager.cc:1347] Generated new cluster ID: b2fc2d37ed1a45c688c857bdb5cceb76
I20260926 06:19:21.696233 23881 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:19:21.700327 23890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:21.693589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:47563 (local address 127.0.0.1:45373)
0926 06:19:21.693757 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:21.693768 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:21.694017 (+ 249us) server_negotiation.cc:407] Connection header received
0926 06:19:21.694568 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:21.694577 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:21.694645 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:21.694787 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:21.695121 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:21.696635 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:21.697849 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:21.698563 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:21.698702 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:21.699523 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:21.699527 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:21.699528 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:19:21.699784 (+ 256us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:21.699830 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:19:21.699848 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:21.699947 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:21.700040 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 06:19:21.700076 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":55,"threads_started":1}
I20260926 06:19:21.701499 2491 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:19:21.707147 23890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:21.703089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:47715 (local address 127.0.0.1:45373)
0926 06:19:21.703131 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:21.703133 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:21.703269 (+ 136us) server_negotiation.cc:407] Connection header received
0926 06:19:21.703338 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:21.703339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:21.703350 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:21.703418 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:21.703570 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:21.704394 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:21.705595 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:21.705936 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:21.705999 (+ 63us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:21.706839 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:21.706841 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:21.706841 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:19:21.706847 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:21.706863 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:19:21.706868 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:21.706928 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:21.707020 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 06:19:21.707035 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 06:19:21.715639 23881 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:19:21.716254 23881 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:19:21.723105 23881 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40b782f443a34b979db4cf7a836c90a3: Generated new TSK 0
I20260926 06:19:21.723320 23881 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:19:21.727614 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 23802
W20260926 06:19:21.733130 23787 connection.cc:535] client connection to 127.0.0.1:45373 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45373): Connection reset by peer (error 104)
2026-09-26T06:19:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1390 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790403292369990-2491-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:19:21 dist-test-slave-hndd krb5kdc[23901](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:19:21 dist-test-slave-hndd krb5kdc[23901](info): set up 2 sockets
Sep 26 06:19:21 dist-test-slave-hndd krb5kdc[23901](info): commencing operation
krb5kdc: starting...
W20260926 06:19:22.813577 2491 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Sep 26 06:19:22 dist-test-slave-hndd krb5kdc[23901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:19:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:19:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790403292369990-2491-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790403292369990-2491-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 06:19:22.940461 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39063
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790403292369990-2491-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39063
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:19:23.021188 23914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:19:23.021442 23914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:19:23.021469 23914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:19:23.022612 23914 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:19:23.022656 23914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:19:23.022671 23914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:19:23.022683 23914 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:19:23.022696 23914 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:19:23.024188 23914 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790403292369990-2491-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39063
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790403292369990-2491-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:19:23.024408 23914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:19:23.024597 23914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:19:23.031358 23919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:23.031354 23920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:23.031384 23922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:23.031407 23914 server_base.cc:1034] running on GCE node
I20260926 06:19:23.031849 23914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:19:23.032163 23914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:19:23.033428 23914 hybrid_clock.cc:648] HybridClock initialized: now 1790403563033400 us; error 47 us; skew 500 ppm
Sep 26 06:19:23 dist-test-slave-hndd krb5kdc[23901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:19:23.036141 23914 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 06:19:23.036679 23914 webserver.cc:460] Webserver started at http://127.0.0.1:44425/ using document root <none> and password file <none>
I20260926 06:19:23.036943 23914 fs_manager.cc:359] Metadata directory not provided
I20260926 06:19:23.037019 23914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:19:23.037184 23914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:19:23.038389 23914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790403292369990-2491-0/minicluster-data/master-0/data/instance:
uuid: "fb06e2fb832d43869feaf3dcd86cee6e"
format_stamp: "Formatted at 2026-09-26 06:19:23 on dist-test-slave-hndd"
server_key: "f305dc5315790f92be0f75d963d674f7"
server_key_iv: "c79b4a218513e7083002c9f5eac41bbc"
server_key_version: "encryptionkey@0"
I20260926 06:19:23.038757 23914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790403292369990-2491-0/minicluster-data/master-0/wal/instance:
uuid: "fb06e2fb832d43869feaf3dcd86cee6e"
format_stamp: "Formatted at 2026-09-26 06:19:23 on dist-test-slave-hndd"
server_key: "f305dc5315790f92be0f75d963d674f7"
server_key_iv: "c79b4a218513e7083002c9f5eac41bbc"
server_key_version: "encryptionkey@0"
I20260926 06:19:23.040189 23914 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:19:23.041030 23929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:23.041298 23914 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:19:23.041615 23914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "fb06e2fb832d43869feaf3dcd86cee6e"
format_stamp: "Formatted at 2026-09-26 06:19:23 on dist-test-slave-hndd"
server_key: "f305dc5315790f92be0f75d963d674f7"
server_key_iv: "c79b4a218513e7083002c9f5eac41bbc"
server_key_version: "encryptionkey@0"
I20260926 06:19:23.041713 23914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:19:23.045433 23914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:19:23.047609 23914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:19:23.047783 23914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:19:23.052094 23914 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39063
I20260926 06:19:23.052126 23981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39063 every 8 connection(s)
I20260926 06:19:23.052481 23914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:19:23.053587 23982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:19:23.055714 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 23914
I20260926 06:19:23.055883 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790403292369990-2491-0/minicluster-data/master-0/wal/instance
I20260926 06:19:23.056107 2491 external_mini_cluster.cc:1428] Setting key d92ff6793f5325b894255ff349fc5edd
I20260926 06:19:23.056595 23982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb06e2fb832d43869feaf3dcd86cee6e: Bootstrap starting.
I20260926 06:19:23.057325 23982 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb06e2fb832d43869feaf3dcd86cee6e: Neither blocks nor log segments found. Creating new log.
I20260926 06:19:23.057754 23982 log.cc:824] T 00000000000000000000000000000000 P fb06e2fb832d43869feaf3dcd86cee6e: Log is configured to *not* fsync() on all Append() calls
I20260926 06:19:23.058656 23982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb06e2fb832d43869feaf3dcd86cee6e: No bootstrap required, opened a new log
I20260926 06:19:23.059196 23985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:23.056696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:44205 (local address 127.0.0.1:39063)
0926 06:19:23.056892 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:23.056899 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:23.056924 (+ 25us) server_negotiation.cc:407] Connection header received
0926 06:19:23.057448 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:23.057473 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:23.057512 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:23.057649 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:23.058524 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:23.058528 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:23.058529 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:19:23.058533 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:23.058580 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:19:23.058605 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:23.058701 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:23.058947 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 06:19:23.058994 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":96,"threads_started":1}
I20260926 06:19:23.060743 2491 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:19:23.061299 23982 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb06e2fb832d43869feaf3dcd86cee6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb06e2fb832d43869feaf3dcd86cee6e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39063 } }
I20260926 06:19:23.061515 23982 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb06e2fb832d43869feaf3dcd86cee6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:19:23.061561 23982 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb06e2fb832d43869feaf3dcd86cee6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb06e2fb832d43869feaf3dcd86cee6e, State: Initialized, Role: FOLLOWER
I20260926 06:19:23.061697 23982 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb06e2fb832d43869feaf3dcd86cee6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb06e2fb832d43869feaf3dcd86cee6e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39063 } }
I20260926 06:19:23.061800 23982 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb06e2fb832d43869feaf3dcd86cee6e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:19:23.061836 23982 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb06e2fb832d43869feaf3dcd86cee6e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:19:23.061885 23982 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb06e2fb832d43869feaf3dcd86cee6e [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:19:23.062738 23982 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb06e2fb832d43869feaf3dcd86cee6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb06e2fb832d43869feaf3dcd86cee6e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39063 } }
I20260926 06:19:23.062880 23982 leader_election.cc:302] T 00000000000000000000000000000000 P fb06e2fb832d43869feaf3dcd86cee6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb06e2fb832d43869feaf3dcd86cee6e; no voters:
I20260926 06:19:23.063063 23982 leader_election.cc:288] T 00000000000000000000000000000000 P fb06e2fb832d43869feaf3dcd86cee6e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:19:23.063124 23992 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb06e2fb832d43869feaf3dcd86cee6e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:19:23.063326 23992 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb06e2fb832d43869feaf3dcd86cee6e [term 1 LEADER]: Becoming Leader. State: Replica: fb06e2fb832d43869feaf3dcd86cee6e, State: Running, Role: LEADER
I20260926 06:19:23.063373 23982 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb06e2fb832d43869feaf3dcd86cee6e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:19:23.063473 23992 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb06e2fb832d43869feaf3dcd86cee6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb06e2fb832d43869feaf3dcd86cee6e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39063 } }
I20260926 06:19:23.063937 23985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:23.062526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:59607 (local address 127.0.0.1:39063)
0926 06:19:23.062572 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:23.062574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:23.062701 (+ 127us) server_negotiation.cc:407] Connection header received
0926 06:19:23.062749 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:23.062750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:23.062756 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:23.062870 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:23.063584 (+ 714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:23.063586 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:23.063586 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:19:23.063591 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:23.063625 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:19:23.063631 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:23.063688 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:23.063835 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 06:19:23.063849 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 06:19:23.063954 23993 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb06e2fb832d43869feaf3dcd86cee6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb06e2fb832d43869feaf3dcd86cee6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb06e2fb832d43869feaf3dcd86cee6e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39063 } } }
I20260926 06:19:23.064214 23993 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb06e2fb832d43869feaf3dcd86cee6e [sys.catalog]: This master's current role is: LEADER
I20260926 06:19:23.064097 23994 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb06e2fb832d43869feaf3dcd86cee6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb06e2fb832d43869feaf3dcd86cee6e. Latest consensus state: current_term: 1 leader_uuid: "fb06e2fb832d43869feaf3dcd86cee6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb06e2fb832d43869feaf3dcd86cee6e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39063 } } }
I20260926 06:19:23.064505 23994 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb06e2fb832d43869feaf3dcd86cee6e [sys.catalog]: This master's current role is: LEADER
I20260926 06:19:23.064687 24001 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:19:23.065299 24001 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:19:23.066920 24001 catalog_manager.cc:1347] Generated new cluster ID: d7760288fb084cd8bfffaed5ec6e8fca
I20260926 06:19:23.066977 24001 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:19:23.075017 24001 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:19:23.075222 24001 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:19:23.081672 24001 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb06e2fb832d43869feaf3dcd86cee6e: Generated new TSK 0
I20260926 06:19:23.081885 24001 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:19:23.083591 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 23914
2026-09-26T06:19:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1355 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790403292369990-2491-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:19:23 dist-test-slave-hndd krb5kdc[24013](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:19:23 dist-test-slave-hndd krb5kdc[24013](info): set up 2 sockets
Sep 26 06:19:23 dist-test-slave-hndd krb5kdc[24013](info): commencing operation
krb5kdc: starting...
W20260926 06:19:24.157799 2491 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Sep 26 06:19:24 dist-test-slave-hndd krb5kdc[24013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:19:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:19:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
I20260926 06:19:24.285849 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:38189
--webserver_interface=127.2.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:44755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.110.254:38189
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:19:24.371070 24026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:19:24.371306 24026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:19:24.371333 24026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:19:24.372700 24026 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:19:24.372754 24026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:19:24.372769 24026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:19:24.372783 24026 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:19:24.372795 24026 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:19:24.374401 24026 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:44755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790403292369990-2491-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.110.254:38189
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:38189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:19:24.374688 24026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:19:24.374933 24026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:19:24.381217 24031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:24.381165 24034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:24.381168 24032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:24.381201 24026 server_base.cc:1034] running on GCE node
I20260926 06:19:24.381637 24026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:19:24.381904 24026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:19:24.383160 24026 hybrid_clock.cc:648] HybridClock initialized: now 1790403564383126 us; error 43 us; skew 500 ppm
Sep 26 06:19:24 dist-test-slave-hndd krb5kdc[24013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403564, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:19:24.391693 24026 init.cc:377] Logged in from keytab as kudu/127.2.110.254@KRBTEST.COM (short username kudu)
I20260926 06:19:24.392268 24026 webserver.cc:460] Webserver started at http://127.2.110.254:34261/ using document root <none> and password file <none>
I20260926 06:19:24.392514 24026 fs_manager.cc:359] Metadata directory not provided
I20260926 06:19:24.392585 24026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:19:24.392750 24026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:19:24.393954 24026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790403292369990-2491-0/minicluster-data/master-0/data/instance:
uuid: "5965205146ae4968bfd9304c08f7a975"
format_stamp: "Formatted at 2026-09-26 06:19:24 on dist-test-slave-hndd"
server_key: "8a17e0b9cbeaedcde2aa0333da433a38"
server_key_iv: "689594a7e5823be54d56196a1f354e28"
server_key_version: "encryptionkey@0"
I20260926 06:19:24.394342 24026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790403292369990-2491-0/minicluster-data/master-0/wal/instance:
uuid: "5965205146ae4968bfd9304c08f7a975"
format_stamp: "Formatted at 2026-09-26 06:19:24 on dist-test-slave-hndd"
server_key: "8a17e0b9cbeaedcde2aa0333da433a38"
server_key_iv: "689594a7e5823be54d56196a1f354e28"
server_key_version: "encryptionkey@0"
I20260926 06:19:24.395936 24026 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:19:24.396855 24041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:24.397174 24026 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:19:24.397306 24026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "5965205146ae4968bfd9304c08f7a975"
format_stamp: "Formatted at 2026-09-26 06:19:24 on dist-test-slave-hndd"
server_key: "8a17e0b9cbeaedcde2aa0333da433a38"
server_key_iv: "689594a7e5823be54d56196a1f354e28"
server_key_version: "encryptionkey@0"
I20260926 06:19:24.397442 24026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:19:24.409274 24026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:19:24.411718 24026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:19:24.411893 24026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:19:24.416251 24026 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.254:38189
I20260926 06:19:24.416499 24093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.254:38189 every 8 connection(s)
I20260926 06:19:24.416710 24026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:19:24.417689 24094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:19:24.419955 24094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5965205146ae4968bfd9304c08f7a975: Bootstrap starting.
I20260926 06:19:24.420499 24094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5965205146ae4968bfd9304c08f7a975: Neither blocks nor log segments found. Creating new log.
I20260926 06:19:24.420733 24094 log.cc:824] T 00000000000000000000000000000000 P 5965205146ae4968bfd9304c08f7a975: Log is configured to *not* fsync() on all Append() calls
I20260926 06:19:24.421500 24094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5965205146ae4968bfd9304c08f7a975: No bootstrap required, opened a new log
I20260926 06:19:24.421653 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 24026
I20260926 06:19:24.421765 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790403292369990-2491-0/minicluster-data/master-0/wal/instance
I20260926 06:19:24.421980 2491 external_mini_cluster.cc:1428] Setting key a03dca93e1c0c7e7c8802919f0691012
I20260926 06:19:24.423789 24094 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5965205146ae4968bfd9304c08f7a975 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5965205146ae4968bfd9304c08f7a975" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 38189 } }
I20260926 06:19:24.423980 24094 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5965205146ae4968bfd9304c08f7a975 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:19:24.424021 24094 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5965205146ae4968bfd9304c08f7a975 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5965205146ae4968bfd9304c08f7a975, State: Initialized, Role: FOLLOWER
I20260926 06:19:24.424197 24094 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5965205146ae4968bfd9304c08f7a975 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5965205146ae4968bfd9304c08f7a975" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 38189 } }
I20260926 06:19:24.424291 24094 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5965205146ae4968bfd9304c08f7a975 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:19:24.424328 24094 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5965205146ae4968bfd9304c08f7a975 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:19:24.424381 24094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5965205146ae4968bfd9304c08f7a975 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:19:24.425523 24094 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5965205146ae4968bfd9304c08f7a975 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5965205146ae4968bfd9304c08f7a975" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 38189 } }
I20260926 06:19:24.425616 24094 leader_election.cc:302] T 00000000000000000000000000000000 P 5965205146ae4968bfd9304c08f7a975 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5965205146ae4968bfd9304c08f7a975; no voters:
I20260926 06:19:24.425860 24094 leader_election.cc:288] T 00000000000000000000000000000000 P 5965205146ae4968bfd9304c08f7a975 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:19:24.425882 24099 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5965205146ae4968bfd9304c08f7a975 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:19:24.425992 24099 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5965205146ae4968bfd9304c08f7a975 [term 1 LEADER]: Becoming Leader. State: Replica: 5965205146ae4968bfd9304c08f7a975, State: Running, Role: LEADER
I20260926 06:19:24.426134 24094 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5965205146ae4968bfd9304c08f7a975 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:19:24.426142 24099 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5965205146ae4968bfd9304c08f7a975 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5965205146ae4968bfd9304c08f7a975" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 38189 } }
I20260926 06:19:24.426640 24101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5965205146ae4968bfd9304c08f7a975 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5965205146ae4968bfd9304c08f7a975. Latest consensus state: current_term: 1 leader_uuid: "5965205146ae4968bfd9304c08f7a975" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5965205146ae4968bfd9304c08f7a975" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 38189 } } }
I20260926 06:19:24.426730 24101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5965205146ae4968bfd9304c08f7a975 [sys.catalog]: This master's current role is: LEADER
I20260926 06:19:24.426952 24100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5965205146ae4968bfd9304c08f7a975 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5965205146ae4968bfd9304c08f7a975" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5965205146ae4968bfd9304c08f7a975" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 38189 } } }
I20260926 06:19:24.427292 24100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5965205146ae4968bfd9304c08f7a975 [sys.catalog]: This master's current role is: LEADER
I20260926 06:19:24.427479 24106 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:19:24.428036 24106 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:19:24.429786 24106 catalog_manager.cc:1347] Generated new cluster ID: 5ce83430d75544cbb86cfbde8db29975
I20260926 06:19:24.429837 24106 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 06:19:24 dist-test-slave-hndd krb5kdc[24013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:19:24.434988 24098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:24.422522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51373 (local address 127.2.110.254:38189)
0926 06:19:24.422712 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:24.422721 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:24.422759 (+ 38us) server_negotiation.cc:407] Connection header received
0926 06:19:24.423759 (+ 1000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:24.423768 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:24.423829 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:24.423959 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:24.424830 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:24.425979 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:24.426999 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:24.427782 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:24.427915 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:24.430859 (+ 2944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:24.430863 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:24.430863 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:19:24.430877 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:24.432937 (+ 2060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:24.433460 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:24.433464 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:24.433465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:24.433598 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:24.433846 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:24.433847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:24.433847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:24.434211 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:24.434391 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:24.434480 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 06:19:24.434587 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":98,"threads_started":1}
I20260926 06:19:24.436304 2491 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:19:24.445505 24098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:24.437830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54691 (local address 127.2.110.254:38189)
0926 06:19:24.437929 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:24.437931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:24.437965 (+ 34us) server_negotiation.cc:407] Connection header received
0926 06:19:24.438027 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:24.438028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:24.438038 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:24.438105 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:24.438719 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:24.439481 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:24.440471 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:24.440819 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:24.440895 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:24.442541 (+ 1646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:24.442543 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:24.442544 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:19:24.442552 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:24.444216 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:24.444628 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:24.444629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:24.444629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:24.444693 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:24.444917 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:24.444918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:24.444918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:24.445061 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:24.445165 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:24.445309 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 06:19:24.445403 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 06:19:24.446945 24106 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:19:24.447674 24106 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:19:24.461711 24106 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5965205146ae4968bfd9304c08f7a975: Generated new TSK 0
I20260926 06:19:24.461953 24106 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:19:24.470562 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 24026
W20260926 06:19:24.475371 24011 connection.cc:535] client connection to 127.2.110.254:38189 recv error: Network error: failed to read from TLS socket (remote: 127.2.110.254:38189): Connection reset by peer (error 104)
2026-09-26T06:19:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1386 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790403292369990-2491-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:19:24 dist-test-slave-hndd krb5kdc[24125](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:19:24 dist-test-slave-hndd krb5kdc[24125](info): set up 2 sockets
Sep 26 06:19:24 dist-test-slave-hndd krb5kdc[24125](info): commencing operation
krb5kdc: starting...
W20260926 06:19:25.529592 2491 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Sep 26 06:19:25 dist-test-slave-hndd krb5kdc[24125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:19:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:19:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
I20260926 06:19:25.658144 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:45529
--webserver_interface=127.2.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:35303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.110.254:45529
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:19:25.740760 24138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:19:25.741032 24138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:19:25.741075 24138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:19:25.742304 24138 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:19:25.742388 24138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:19:25.742413 24138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:19:25.742434 24138 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:19:25.742463 24138 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:19:25.744009 24138 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:35303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790403292369990-2491-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.110.254:45529
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:45529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:19:25.744304 24138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:19:25.744544 24138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:19:25.749794 24146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:25.749797 24143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:25.749856 24138 server_base.cc:1034] running on GCE node
W20260926 06:19:25.750044 24144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:25.750285 24138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:19:25.750587 24138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:19:25.751758 24138 hybrid_clock.cc:648] HybridClock initialized: now 1790403565751759 us; error 44 us; skew 500 ppm
Sep 26 06:19:25 dist-test-slave-hndd krb5kdc[24125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403565, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:19:25.754443 24138 init.cc:377] Logged in from keytab as kudu/127.2.110.254@KRBTEST.COM (short username kudu)
I20260926 06:19:25.754917 24138 webserver.cc:460] Webserver started at http://127.2.110.254:34941/ using document root <none> and password file <none>
I20260926 06:19:25.755160 24138 fs_manager.cc:359] Metadata directory not provided
I20260926 06:19:25.755231 24138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:19:25.755378 24138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:19:25.756472 24138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790403292369990-2491-0/minicluster-data/master-0/data/instance:
uuid: "5e08d684e88c4dfbaf610410bfeab5fb"
format_stamp: "Formatted at 2026-09-26 06:19:25 on dist-test-slave-hndd"
server_key: "6d86d0c6ccbbc5c509f95647c316d14a"
server_key_iv: "c90c2287fc53ccae3f4e52dc9c111bba"
server_key_version: "encryptionkey@0"
I20260926 06:19:25.756836 24138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790403292369990-2491-0/minicluster-data/master-0/wal/instance:
uuid: "5e08d684e88c4dfbaf610410bfeab5fb"
format_stamp: "Formatted at 2026-09-26 06:19:25 on dist-test-slave-hndd"
server_key: "6d86d0c6ccbbc5c509f95647c316d14a"
server_key_iv: "c90c2287fc53ccae3f4e52dc9c111bba"
server_key_version: "encryptionkey@0"
I20260926 06:19:25.758260 24138 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 06:19:25.758977 24153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:25.759318 24138 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:19:25.759498 24138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "5e08d684e88c4dfbaf610410bfeab5fb"
format_stamp: "Formatted at 2026-09-26 06:19:25 on dist-test-slave-hndd"
server_key: "6d86d0c6ccbbc5c509f95647c316d14a"
server_key_iv: "c90c2287fc53ccae3f4e52dc9c111bba"
server_key_version: "encryptionkey@0"
I20260926 06:19:25.759627 24138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:19:25.775352 24138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:19:25.777944 24138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:19:25.778116 24138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:19:25.782130 24138 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.254:45529
I20260926 06:19:25.782196 24205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.254:45529 every 8 connection(s)
I20260926 06:19:25.782541 24138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:19:25.783461 24206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:19:25.783561 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 24138
I20260926 06:19:25.783669 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790403292369990-2491-0/minicluster-data/master-0/wal/instance
I20260926 06:19:25.783879 2491 external_mini_cluster.cc:1428] Setting key 47acfaece691efef23d37c6de93cfb60
I20260926 06:19:25.786890 24206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e08d684e88c4dfbaf610410bfeab5fb: Bootstrap starting.
I20260926 06:19:25.787591 24206 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e08d684e88c4dfbaf610410bfeab5fb: Neither blocks nor log segments found. Creating new log.
I20260926 06:19:25.787905 24206 log.cc:824] T 00000000000000000000000000000000 P 5e08d684e88c4dfbaf610410bfeab5fb: Log is configured to *not* fsync() on all Append() calls
I20260926 06:19:25.788695 24206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e08d684e88c4dfbaf610410bfeab5fb: No bootstrap required, opened a new log
I20260926 06:19:25.789767 24209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:25.784435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57559 (local address 127.2.110.254:45529)
0926 06:19:25.784612 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:25.784619 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:25.784644 (+ 25us) server_negotiation.cc:407] Connection header received
0926 06:19:25.785123 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:25.785130 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:25.785176 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:25.785307 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:25.785610 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:25.786512 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:25.787570 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:25.788025 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:25.788116 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:25.788961 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:25.788965 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:25.788965 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:19:25.789173 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:25.789212 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:19:25.789228 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:25.789323 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:25.789491 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 06:19:25.789522 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":86,"threads_started":1}
I20260926 06:19:25.791020 2491 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:19:25.791286 24206 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e08d684e88c4dfbaf610410bfeab5fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e08d684e88c4dfbaf610410bfeab5fb" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 45529 } }
I20260926 06:19:25.791458 24206 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e08d684e88c4dfbaf610410bfeab5fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:19:25.791497 24206 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e08d684e88c4dfbaf610410bfeab5fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e08d684e88c4dfbaf610410bfeab5fb, State: Initialized, Role: FOLLOWER
I20260926 06:19:25.791633 24206 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e08d684e88c4dfbaf610410bfeab5fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e08d684e88c4dfbaf610410bfeab5fb" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 45529 } }
I20260926 06:19:25.791709 24206 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e08d684e88c4dfbaf610410bfeab5fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:19:25.791743 24206 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e08d684e88c4dfbaf610410bfeab5fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:19:25.791802 24206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e08d684e88c4dfbaf610410bfeab5fb [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:19:25.792641 24206 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e08d684e88c4dfbaf610410bfeab5fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e08d684e88c4dfbaf610410bfeab5fb" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 45529 } }
I20260926 06:19:25.792762 24206 leader_election.cc:302] T 00000000000000000000000000000000 P 5e08d684e88c4dfbaf610410bfeab5fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e08d684e88c4dfbaf610410bfeab5fb; no voters:
I20260926 06:19:25.792982 24206 leader_election.cc:288] T 00000000000000000000000000000000 P 5e08d684e88c4dfbaf610410bfeab5fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:19:25.793042 24216 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e08d684e88c4dfbaf610410bfeab5fb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:19:25.793195 24216 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e08d684e88c4dfbaf610410bfeab5fb [term 1 LEADER]: Becoming Leader. State: Replica: 5e08d684e88c4dfbaf610410bfeab5fb, State: Running, Role: LEADER
I20260926 06:19:25.793268 24206 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e08d684e88c4dfbaf610410bfeab5fb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:19:25.793309 24216 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e08d684e88c4dfbaf610410bfeab5fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e08d684e88c4dfbaf610410bfeab5fb" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 45529 } }
I20260926 06:19:25.793777 24217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e08d684e88c4dfbaf610410bfeab5fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e08d684e88c4dfbaf610410bfeab5fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e08d684e88c4dfbaf610410bfeab5fb" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 45529 } } }
I20260926 06:19:25.793887 24217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e08d684e88c4dfbaf610410bfeab5fb [sys.catalog]: This master's current role is: LEADER
I20260926 06:19:25.793787 24218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e08d684e88c4dfbaf610410bfeab5fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e08d684e88c4dfbaf610410bfeab5fb. Latest consensus state: current_term: 1 leader_uuid: "5e08d684e88c4dfbaf610410bfeab5fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e08d684e88c4dfbaf610410bfeab5fb" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 45529 } } }
I20260926 06:19:25.794093 24218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e08d684e88c4dfbaf610410bfeab5fb [sys.catalog]: This master's current role is: LEADER
W20260926 06:19:25.795188 24231 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5e08d684e88c4dfbaf610410bfeab5fb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 06:19:25.795284 24231 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 06:19:25.795370 24225 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:19:25.795583 24225 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:19:25.796520 24209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:25.792623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44027 (local address 127.2.110.254:45529)
0926 06:19:25.792667 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:25.792669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:25.792832 (+ 163us) server_negotiation.cc:407] Connection header received
0926 06:19:25.792859 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:25.792860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:25.792872 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:25.792940 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:25.793129 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:25.793963 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:25.795098 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:25.795434 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:25.795508 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:25.796296 (+ 788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:25.796297 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:25.796297 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:19:25.796302 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:25.796318 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:19:25.796322 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:25.796374 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:25.796435 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 06:19:25.796445 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 06:19:25.797165 24225 catalog_manager.cc:1347] Generated new cluster ID: 83ed83cc444841928b7b4a92b83ad2ab
I20260926 06:19:25.797223 24225 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:19:25.814031 24225 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:19:25.814616 24225 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:19:25.821151 24225 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e08d684e88c4dfbaf610410bfeab5fb: Generated new TSK 0
I20260926 06:19:25.821396 24225 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:19:25.832789 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 24138
W20260926 06:19:25.837311 24123 connection.cc:535] client connection to 127.2.110.254:45529 recv error: Network error: failed to read from TLS socket (remote: 127.2.110.254:45529): Connection reset by peer (error 104)
2026-09-26T06:19:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1362 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790403292369990-2491-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:19:25 dist-test-slave-hndd krb5kdc[24237](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:19:25 dist-test-slave-hndd krb5kdc[24237](info): set up 2 sockets
Sep 26 06:19:25 dist-test-slave-hndd krb5kdc[24237](info): commencing operation
krb5kdc: starting...
W20260926 06:19:26.901726 2491 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Sep 26 06:19:26 dist-test-slave-hndd krb5kdc[24237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:19:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:19:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
I20260926 06:19:27.028712 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:35705
--webserver_interface=127.2.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:43373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.110.254:35705
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:19:27.115661 24250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:19:27.115868 24250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:19:27.115891 24250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:19:27.117062 24250 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:19:27.117125 24250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:19:27.117142 24250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:19:27.117156 24250 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:19:27.117167 24250 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:19:27.118701 24250 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:43373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790403292369990-2491-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.110.254:35705
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:35705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:19:27.118952 24250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:19:27.119203 24250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:19:27.124567 24255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:27.124591 24258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:27.124598 24256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:27.124859 24250 server_base.cc:1034] running on GCE node
I20260926 06:19:27.125082 24250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:19:27.125453 24250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:19:27.126597 24250 hybrid_clock.cc:648] HybridClock initialized: now 1790403567126583 us; error 31 us; skew 500 ppm
Sep 26 06:19:27 dist-test-slave-hndd krb5kdc[24237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403567, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:19:27.129168 24250 init.cc:377] Logged in from keytab as kudu/127.2.110.254@KRBTEST.COM (short username kudu)
I20260926 06:19:27.129631 24250 webserver.cc:460] Webserver started at http://127.2.110.254:40435/ using document root <none> and password file <none>
I20260926 06:19:27.129853 24250 fs_manager.cc:359] Metadata directory not provided
I20260926 06:19:27.129894 24250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:19:27.129981 24250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:19:27.130991 24250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790403292369990-2491-0/minicluster-data/master-0/data/instance:
uuid: "a6837ad440234d669cfaa282a683f820"
format_stamp: "Formatted at 2026-09-26 06:19:27 on dist-test-slave-hndd"
server_key: "af95ab8cb5e9c362dc6e03f21ca8fd93"
server_key_iv: "8d59ce940b2a9b4f3b534eff03ccc286"
server_key_version: "encryptionkey@0"
I20260926 06:19:27.131295 24250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790403292369990-2491-0/minicluster-data/master-0/wal/instance:
uuid: "a6837ad440234d669cfaa282a683f820"
format_stamp: "Formatted at 2026-09-26 06:19:27 on dist-test-slave-hndd"
server_key: "af95ab8cb5e9c362dc6e03f21ca8fd93"
server_key_iv: "8d59ce940b2a9b4f3b534eff03ccc286"
server_key_version: "encryptionkey@0"
I20260926 06:19:27.132853 24250 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 06:19:27.133589 24265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:27.133771 24250 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 06:19:27.133870 24250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "a6837ad440234d669cfaa282a683f820"
format_stamp: "Formatted at 2026-09-26 06:19:27 on dist-test-slave-hndd"
server_key: "af95ab8cb5e9c362dc6e03f21ca8fd93"
server_key_iv: "8d59ce940b2a9b4f3b534eff03ccc286"
server_key_version: "encryptionkey@0"
I20260926 06:19:27.133940 24250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:19:27.137449 24250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:19:27.139665 24250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:19:27.139809 24250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:19:27.143962 24250 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.254:35705
I20260926 06:19:27.144018 24317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.254:35705 every 8 connection(s)
I20260926 06:19:27.144627 24250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:19:27.145828 24318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:19:27.148036 24318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6837ad440234d669cfaa282a683f820: Bootstrap starting.
I20260926 06:19:27.148605 24318 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6837ad440234d669cfaa282a683f820: Neither blocks nor log segments found. Creating new log.
I20260926 06:19:27.148908 24318 log.cc:824] T 00000000000000000000000000000000 P a6837ad440234d669cfaa282a683f820: Log is configured to *not* fsync() on all Append() calls
I20260926 06:19:27.149976 24318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6837ad440234d669cfaa282a683f820: No bootstrap required, opened a new log
I20260926 06:19:27.151629 24318 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6837ad440234d669cfaa282a683f820 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6837ad440234d669cfaa282a683f820" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 35705 } }
I20260926 06:19:27.151782 24318 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6837ad440234d669cfaa282a683f820 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:19:27.151804 24318 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6837ad440234d669cfaa282a683f820 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6837ad440234d669cfaa282a683f820, State: Initialized, Role: FOLLOWER
I20260926 06:19:27.151973 24318 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6837ad440234d669cfaa282a683f820 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6837ad440234d669cfaa282a683f820" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 35705 } }
I20260926 06:19:27.152040 24318 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6837ad440234d669cfaa282a683f820 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:19:27.152220 24318 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6837ad440234d669cfaa282a683f820 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:19:27.152343 24318 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6837ad440234d669cfaa282a683f820 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:19:27.153431 24318 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6837ad440234d669cfaa282a683f820 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6837ad440234d669cfaa282a683f820" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 35705 } }
I20260926 06:19:27.153576 24318 leader_election.cc:302] T 00000000000000000000000000000000 P a6837ad440234d669cfaa282a683f820 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6837ad440234d669cfaa282a683f820; no voters:
I20260926 06:19:27.153796 24318 leader_election.cc:288] T 00000000000000000000000000000000 P a6837ad440234d669cfaa282a683f820 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:19:27.153822 24321 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6837ad440234d669cfaa282a683f820 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:19:27.154031 24321 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6837ad440234d669cfaa282a683f820 [term 1 LEADER]: Becoming Leader. State: Replica: a6837ad440234d669cfaa282a683f820, State: Running, Role: LEADER
I20260926 06:19:27.154119 24318 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6837ad440234d669cfaa282a683f820 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:19:27.154173 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 24250
I20260926 06:19:27.154186 24321 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6837ad440234d669cfaa282a683f820 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6837ad440234d669cfaa282a683f820" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 35705 } }
I20260926 06:19:27.154310 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790403292369990-2491-0/minicluster-data/master-0/wal/instance
I20260926 06:19:27.154513 2491 external_mini_cluster.cc:1428] Setting key 85bf81a69fc3e948f64429d83682d7b9
I20260926 06:19:27.154563 24321 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6837ad440234d669cfaa282a683f820 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6837ad440234d669cfaa282a683f820. Latest consensus state: current_term: 1 leader_uuid: "a6837ad440234d669cfaa282a683f820" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6837ad440234d669cfaa282a683f820" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 35705 } } }
I20260926 06:19:27.154624 24321 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6837ad440234d669cfaa282a683f820 [sys.catalog]: This master's current role is: LEADER
I20260926 06:19:27.154652 24322 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6837ad440234d669cfaa282a683f820 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6837ad440234d669cfaa282a683f820" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6837ad440234d669cfaa282a683f820" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 35705 } } }
I20260926 06:19:27.155169 24322 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6837ad440234d669cfaa282a683f820 [sys.catalog]: This master's current role is: LEADER
W20260926 06:19:27.156213 24339 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a6837ad440234d669cfaa282a683f820: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 06:19:27.156308 24339 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 06:19:27.156378 24334 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:19:27.156544 24334 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:19:27.157294 24331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:27.155038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50083 (local address 127.2.110.254:35705)
0926 06:19:27.155582 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:27.155598 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:27.155625 (+ 27us) server_negotiation.cc:407] Connection header received
0926 06:19:27.155763 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:27.155771 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:27.155895 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:27.156054 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:27.156807 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:27.156811 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:27.156811 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:19:27.156815 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:27.156864 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:19:27.156876 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:27.156957 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:27.157026 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 06:19:27.157071 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":84,"threads_started":1}
I20260926 06:19:27.158335 24334 catalog_manager.cc:1347] Generated new cluster ID: aa46294da3b748cc9d212f168ae168f9
I20260926 06:19:27.158388 24334 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:19:27.158389 2491 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:19:27.163615 24331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:27.161966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52957 (local address 127.2.110.254:35705)
0926 06:19:27.162030 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:27.162033 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:27.162230 (+ 197us) server_negotiation.cc:407] Connection header received
0926 06:19:27.162321 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:27.162322 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:27.162332 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:27.162391 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:27.163323 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:27.163325 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:27.163326 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:19:27.163332 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:27.163354 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:19:27.163361 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:27.163444 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:27.163503 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 06:19:27.163522 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 06:19:27.184095 24334 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:19:27.184304 24334 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:19:27.195516 24334 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6837ad440234d669cfaa282a683f820: Generated new TSK 0
I20260926 06:19:27.195757 24334 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:19:27.197957 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 24250
2026-09-26T06:19:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1365 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790403292369990-2491-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:19:27 dist-test-slave-hndd krb5kdc[24349](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:19:27 dist-test-slave-hndd krb5kdc[24349](info): set up 2 sockets
Sep 26 06:19:27 dist-test-slave-hndd krb5kdc[24349](info): commencing operation
krb5kdc: starting...
W20260926 06:19:28.241860 2491 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Sep 26 06:19:28 dist-test-slave-hndd krb5kdc[24349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:19:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:19:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
I20260926 06:19:28.377034 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:39793
--webserver_interface=127.2.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:42755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.110.254:39793
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:19:28.460189 24362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:19:28.460443 24362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:19:28.460482 24362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:19:28.461683 24362 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:19:28.461760 24362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:19:28.461789 24362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:19:28.461822 24362 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:19:28.461846 24362 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:19:28.463364 24362 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:42755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790403292369990-2491-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.110.254:39793
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:39793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:19:28.463660 24362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:19:28.463893 24362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:19:28.468868 24370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:28.468946 24367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:28.468879 24368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:28.469221 24362 server_base.cc:1034] running on GCE node
I20260926 06:19:28.469640 24362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:19:28.469961 24362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:19:28.471123 24362 hybrid_clock.cc:648] HybridClock initialized: now 1790403568471097 us; error 46 us; skew 500 ppm
Sep 26 06:19:28 dist-test-slave-hndd krb5kdc[24349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403568, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:19:28.473909 24362 init.cc:377] Logged in from keytab as kudu/127.2.110.254@KRBTEST.COM (short username kudu)
I20260926 06:19:28.474422 24362 webserver.cc:460] Webserver started at http://127.2.110.254:34261/ using document root <none> and password file <none>
I20260926 06:19:28.474653 24362 fs_manager.cc:359] Metadata directory not provided
I20260926 06:19:28.474707 24362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:19:28.474841 24362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:19:28.475925 24362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790403292369990-2491-0/minicluster-data/master-0/data/instance:
uuid: "0e541dbb09164b1db20fbaf66e17531b"
format_stamp: "Formatted at 2026-09-26 06:19:28 on dist-test-slave-hndd"
server_key: "73025b64cd0db0f1249b5e4b7480e155"
server_key_iv: "61221e0bc45fd168b0827ca0c1e6c6c5"
server_key_version: "encryptionkey@0"
I20260926 06:19:28.476290 24362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790403292369990-2491-0/minicluster-data/master-0/wal/instance:
uuid: "0e541dbb09164b1db20fbaf66e17531b"
format_stamp: "Formatted at 2026-09-26 06:19:28 on dist-test-slave-hndd"
server_key: "73025b64cd0db0f1249b5e4b7480e155"
server_key_iv: "61221e0bc45fd168b0827ca0c1e6c6c5"
server_key_version: "encryptionkey@0"
I20260926 06:19:28.477712 24362 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:19:28.478509 24377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:28.478694 24362 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:19:28.478798 24362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "0e541dbb09164b1db20fbaf66e17531b"
format_stamp: "Formatted at 2026-09-26 06:19:28 on dist-test-slave-hndd"
server_key: "73025b64cd0db0f1249b5e4b7480e155"
server_key_iv: "61221e0bc45fd168b0827ca0c1e6c6c5"
server_key_version: "encryptionkey@0"
I20260926 06:19:28.478883 24362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:19:28.493681 24362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:19:28.496170 24362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:19:28.496352 24362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:19:28.500610 24362 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.254:39793
I20260926 06:19:28.500644 24429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.254:39793 every 8 connection(s)
I20260926 06:19:28.501000 24362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:19:28.501940 24430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:19:28.502703 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 24362
I20260926 06:19:28.502840 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790403292369990-2491-0/minicluster-data/master-0/wal/instance
I20260926 06:19:28.503039 2491 external_mini_cluster.cc:1428] Setting key 5928714ee7279adb0eb174615eaacb7f
I20260926 06:19:28.505616 24430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e541dbb09164b1db20fbaf66e17531b: Bootstrap starting.
I20260926 06:19:28.506459 24430 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e541dbb09164b1db20fbaf66e17531b: Neither blocks nor log segments found. Creating new log.
I20260926 06:19:28.506938 24430 log.cc:824] T 00000000000000000000000000000000 P 0e541dbb09164b1db20fbaf66e17531b: Log is configured to *not* fsync() on all Append() calls
I20260926 06:19:28.507755 24430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e541dbb09164b1db20fbaf66e17531b: No bootstrap required, opened a new log
I20260926 06:19:28.509907 24430 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e541dbb09164b1db20fbaf66e17531b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e541dbb09164b1db20fbaf66e17531b" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 39793 } }
I20260926 06:19:28.510046 24430 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e541dbb09164b1db20fbaf66e17531b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:19:28.510079 24430 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e541dbb09164b1db20fbaf66e17531b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e541dbb09164b1db20fbaf66e17531b, State: Initialized, Role: FOLLOWER
I20260926 06:19:28.510237 24430 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e541dbb09164b1db20fbaf66e17531b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e541dbb09164b1db20fbaf66e17531b" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 39793 } }
Sep 26 06:19:28 dist-test-slave-hndd krb5kdc[24349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:19:28.510303 24430 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e541dbb09164b1db20fbaf66e17531b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:19:28.510360 24430 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e541dbb09164b1db20fbaf66e17531b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:19:28.510416 24430 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e541dbb09164b1db20fbaf66e17531b [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:19:28.511088 24430 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e541dbb09164b1db20fbaf66e17531b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e541dbb09164b1db20fbaf66e17531b" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 39793 } }
I20260926 06:19:28.511178 24430 leader_election.cc:302] T 00000000000000000000000000000000 P 0e541dbb09164b1db20fbaf66e17531b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e541dbb09164b1db20fbaf66e17531b; no voters:
I20260926 06:19:28.511404 24430 leader_election.cc:288] T 00000000000000000000000000000000 P 0e541dbb09164b1db20fbaf66e17531b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:19:28.511442 24435 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e541dbb09164b1db20fbaf66e17531b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:19:28.511559 24435 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e541dbb09164b1db20fbaf66e17531b [term 1 LEADER]: Becoming Leader. State: Replica: 0e541dbb09164b1db20fbaf66e17531b, State: Running, Role: LEADER
I20260926 06:19:28.511701 24430 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e541dbb09164b1db20fbaf66e17531b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:19:28.511704 24435 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e541dbb09164b1db20fbaf66e17531b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e541dbb09164b1db20fbaf66e17531b" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 39793 } }
I20260926 06:19:28.512156 24436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e541dbb09164b1db20fbaf66e17531b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e541dbb09164b1db20fbaf66e17531b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e541dbb09164b1db20fbaf66e17531b" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 39793 } } }
I20260926 06:19:28.512253 24436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e541dbb09164b1db20fbaf66e17531b [sys.catalog]: This master's current role is: LEADER
I20260926 06:19:28.512528 24442 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:19:28.513062 24442 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:19:28.512432 24437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e541dbb09164b1db20fbaf66e17531b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e541dbb09164b1db20fbaf66e17531b. Latest consensus state: current_term: 1 leader_uuid: "0e541dbb09164b1db20fbaf66e17531b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e541dbb09164b1db20fbaf66e17531b" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 39793 } } }
I20260926 06:19:28.513594 24437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e541dbb09164b1db20fbaf66e17531b [sys.catalog]: This master's current role is: LEADER
I20260926 06:19:28.514889 24433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:28.503613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48721 (local address 127.2.110.254:39793)
0926 06:19:28.503804 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:28.503811 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:28.503836 (+ 25us) server_negotiation.cc:407] Connection header received
0926 06:19:28.504253 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:28.504259 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:28.504306 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:28.504428 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:28.505380 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:28.506313 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:28.507564 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:28.508153 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:28.508297 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:28.510807 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:28.510811 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:28.510811 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:19:28.510824 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:28.513158 (+ 2334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:28.513648 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:28.513651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:28.513652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:28.513762 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:28.513985 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:28.513986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:28.513986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:28.514303 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:28.514434 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:28.514548 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 06:19:28.514683 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":99,"threads_started":1}
I20260926 06:19:28.516350 2491 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:19:28.522581 24442 catalog_manager.cc:1347] Generated new cluster ID: 5bfd2da9487a440485995338a5d029ec
I20260926 06:19:28.522677 24442 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:19:28.526087 24433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:28.517835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53817 (local address 127.2.110.254:39793)
0926 06:19:28.517896 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:28.517898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:28.518102 (+ 204us) server_negotiation.cc:407] Connection header received
0926 06:19:28.518257 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:28.518258 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:28.518266 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:28.518322 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:28.519184 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:28.519686 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:28.520760 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:28.520982 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:28.521044 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:28.522660 (+ 1616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:28.522661 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:28.522662 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:19:28.522670 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:28.524495 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:28.524909 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:28.524909 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:28.524910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:28.524968 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:28.525475 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:28.525479 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:28.525480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:28.525639 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:28.525762 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:28.525916 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 06:19:28.525997 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 06:19:28.535784 24442 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:19:28.536365 24442 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:19:28.545047 24442 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e541dbb09164b1db20fbaf66e17531b: Generated new TSK 0
I20260926 06:19:28.545388 24442 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:19:28.557160 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 24362
2026-09-26T06:19:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1359 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790403292369990-2491-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:19:28 dist-test-slave-hndd krb5kdc[24461](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:19:28 dist-test-slave-hndd krb5kdc[24461](info): set up 2 sockets
Sep 26 06:19:28 dist-test-slave-hndd krb5kdc[24461](info): commencing operation
krb5kdc: starting...
W20260926 06:19:29.654131 2491 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Sep 26 06:19:29 dist-test-slave-hndd krb5kdc[24461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:19:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:19:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
I20260926 06:19:29.788154 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:34265
--webserver_interface=127.2.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:43487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.110.254:34265
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:19:29.872267 24474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:19:29.872486 24474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:19:29.872511 24474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:19:29.873840 24474 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:19:29.873888 24474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:19:29.873903 24474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:19:29.873915 24474 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:19:29.873929 24474 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:19:29.875470 24474 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:43487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790403292369990-2491-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.110.254:34265
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:34265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:19:29.875739 24474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:19:29.876001 24474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:19:29.880932 24482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:29.880952 24480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:29.880990 24479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:29.881184 24474 server_base.cc:1034] running on GCE node
I20260926 06:19:29.881481 24474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:19:29.881786 24474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:19:29.882969 24474 hybrid_clock.cc:648] HybridClock initialized: now 1790403569882953 us; error 33 us; skew 500 ppm
Sep 26 06:19:29 dist-test-slave-hndd krb5kdc[24461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403569, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:19:29.889215 24474 init.cc:377] Logged in from keytab as kudu/127.2.110.254@KRBTEST.COM (short username kudu)
I20260926 06:19:29.889796 24474 webserver.cc:460] Webserver started at http://127.2.110.254:35077/ using document root <none> and password file <none>
I20260926 06:19:29.890053 24474 fs_manager.cc:359] Metadata directory not provided
I20260926 06:19:29.890125 24474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:19:29.890280 24474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:19:29.891397 24474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790403292369990-2491-0/minicluster-data/master-0/data/instance:
uuid: "63824de614fe4e33bf7c207a30ff9dfe"
format_stamp: "Formatted at 2026-09-26 06:19:29 on dist-test-slave-hndd"
server_key: "ec7c98fe383fca43832dbe5f668b7353"
server_key_iv: "c2c80cb4797cf53a55dc23c38dd0061a"
server_key_version: "encryptionkey@0"
I20260926 06:19:29.891774 24474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790403292369990-2491-0/minicluster-data/master-0/wal/instance:
uuid: "63824de614fe4e33bf7c207a30ff9dfe"
format_stamp: "Formatted at 2026-09-26 06:19:29 on dist-test-slave-hndd"
server_key: "ec7c98fe383fca43832dbe5f668b7353"
server_key_iv: "c2c80cb4797cf53a55dc23c38dd0061a"
server_key_version: "encryptionkey@0"
I20260926 06:19:29.893211 24474 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:19:29.894235 24489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:29.894557 24474 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:19:29.894704 24474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "63824de614fe4e33bf7c207a30ff9dfe"
format_stamp: "Formatted at 2026-09-26 06:19:29 on dist-test-slave-hndd"
server_key: "ec7c98fe383fca43832dbe5f668b7353"
server_key_iv: "c2c80cb4797cf53a55dc23c38dd0061a"
server_key_version: "encryptionkey@0"
I20260926 06:19:29.894789 24474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:19:29.921439 24474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:19:29.923818 24474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:19:29.923992 24474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:19:29.928201 24474 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.254:34265
I20260926 06:19:29.928237 24541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.254:34265 every 8 connection(s)
I20260926 06:19:29.928633 24474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:19:29.929584 24542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:19:29.931869 24542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63824de614fe4e33bf7c207a30ff9dfe: Bootstrap starting.
I20260926 06:19:29.932472 24542 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63824de614fe4e33bf7c207a30ff9dfe: Neither blocks nor log segments found. Creating new log.
I20260926 06:19:29.932771 24542 log.cc:824] T 00000000000000000000000000000000 P 63824de614fe4e33bf7c207a30ff9dfe: Log is configured to *not* fsync() on all Append() calls
I20260926 06:19:29.933550 24542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63824de614fe4e33bf7c207a30ff9dfe: No bootstrap required, opened a new log
I20260926 06:19:29.933816 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 24474
I20260926 06:19:29.933951 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790403292369990-2491-0/minicluster-data/master-0/wal/instance
I20260926 06:19:29.934139 2491 external_mini_cluster.cc:1428] Setting key c656b2d41215e069a90794754ca15979
I20260926 06:19:29.936323 24542 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63824de614fe4e33bf7c207a30ff9dfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63824de614fe4e33bf7c207a30ff9dfe" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 34265 } }
I20260926 06:19:29.936506 24542 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63824de614fe4e33bf7c207a30ff9dfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:19:29.936548 24542 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63824de614fe4e33bf7c207a30ff9dfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63824de614fe4e33bf7c207a30ff9dfe, State: Initialized, Role: FOLLOWER
I20260926 06:19:29.936704 24542 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63824de614fe4e33bf7c207a30ff9dfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63824de614fe4e33bf7c207a30ff9dfe" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 34265 } }
I20260926 06:19:29.936810 24542 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63824de614fe4e33bf7c207a30ff9dfe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:19:29.936843 24542 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63824de614fe4e33bf7c207a30ff9dfe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:19:29.936882 24542 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63824de614fe4e33bf7c207a30ff9dfe [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:19:29.937642 24542 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63824de614fe4e33bf7c207a30ff9dfe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63824de614fe4e33bf7c207a30ff9dfe" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 34265 } }
I20260926 06:19:29.937752 24542 leader_election.cc:302] T 00000000000000000000000000000000 P 63824de614fe4e33bf7c207a30ff9dfe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63824de614fe4e33bf7c207a30ff9dfe; no voters:
I20260926 06:19:29.937896 24542 leader_election.cc:288] T 00000000000000000000000000000000 P 63824de614fe4e33bf7c207a30ff9dfe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:19:29.938006 24547 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63824de614fe4e33bf7c207a30ff9dfe [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:19:29.938210 24547 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63824de614fe4e33bf7c207a30ff9dfe [term 1 LEADER]: Becoming Leader. State: Replica: 63824de614fe4e33bf7c207a30ff9dfe, State: Running, Role: LEADER
I20260926 06:19:29.938243 24542 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63824de614fe4e33bf7c207a30ff9dfe [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:19:29.939455 24547 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63824de614fe4e33bf7c207a30ff9dfe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63824de614fe4e33bf7c207a30ff9dfe" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 34265 } }
I20260926 06:19:29.940006 24549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63824de614fe4e33bf7c207a30ff9dfe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63824de614fe4e33bf7c207a30ff9dfe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63824de614fe4e33bf7c207a30ff9dfe" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 34265 } } }
I20260926 06:19:29.940469 24549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63824de614fe4e33bf7c207a30ff9dfe [sys.catalog]: This master's current role is: LEADER
I20260926 06:19:29.940697 24563 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:19:29.940932 24548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63824de614fe4e33bf7c207a30ff9dfe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63824de614fe4e33bf7c207a30ff9dfe. Latest consensus state: current_term: 1 leader_uuid: "63824de614fe4e33bf7c207a30ff9dfe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63824de614fe4e33bf7c207a30ff9dfe" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 34265 } } }
I20260926 06:19:29.941066 24548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63824de614fe4e33bf7c207a30ff9dfe [sys.catalog]: This master's current role is: LEADER
I20260926 06:19:29.941429 24563 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:19:29.941663 24546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:29.934676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58119 (local address 127.2.110.254:34265)
0926 06:19:29.934879 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:29.934889 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:29.934929 (+ 40us) server_negotiation.cc:407] Connection header received
0926 06:19:29.936273 (+ 1344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:29.936282 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:29.936346 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:29.936502 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:29.936805 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:29.938185 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:29.939151 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:29.939699 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:29.939836 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:29.940628 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:29.940631 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:29.940632 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:19:29.940872 (+ 240us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:29.940920 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:19:29.940937 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:29.941296 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:29.941332 (+ 36us) server_negotiation.cc:299] Negotiation successful
0926 06:19:29.941427 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":88,"threads_started":1}
I20260926 06:19:29.942617 2491 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:19:29.943017 24563 catalog_manager.cc:1347] Generated new cluster ID: 0c3c4c7773e44e219806ee4bda4bdcd9
I20260926 06:19:29.943056 24563 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:19:29.948823 24546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:29.944119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54449 (local address 127.2.110.254:34265)
0926 06:19:29.944153 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:29.944155 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:29.945508 (+ 1353us) server_negotiation.cc:407] Connection header received
0926 06:19:29.945557 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:29.945558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:29.945570 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:29.945630 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:29.945755 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:29.946490 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:29.947473 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:29.947780 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:29.947858 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:29.948565 (+ 707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:29.948566 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:29.948566 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:19:29.948572 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:29.948590 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:19:29.948595 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:29.948656 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:29.948712 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 06:19:29.948740 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 06:19:29.954034 24563 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:19:29.954597 24563 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:19:29.965301 24563 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63824de614fe4e33bf7c207a30ff9dfe: Generated new TSK 0
I20260926 06:19:29.965566 24563 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:19:29.967906 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 24474
2026-09-26T06:19:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1409 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790403292369990-2491-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:19:30 dist-test-slave-hndd krb5kdc[24574](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:19:30 dist-test-slave-hndd krb5kdc[24574](info): set up 2 sockets
Sep 26 06:19:30 dist-test-slave-hndd krb5kdc[24574](info): commencing operation
krb5kdc: starting...
W20260926 06:19:31.025692 2491 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Sep 26 06:19:31 dist-test-slave-hndd krb5kdc[24574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:19:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:19:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
I20260926 06:19:31.152149 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:33539
--webserver_interface=127.2.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:44005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.110.254:33539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:19:31.238332 24587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:19:31.238603 24587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:19:31.238647 24587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:19:31.239909 24587 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:19:31.239957 24587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:19:31.240007 24587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:19:31.240031 24587 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:19:31.240057 24587 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:19:31.241686 24587 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:44005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790403292369990-2491-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.110.254:33539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:33539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:19:31.241995 24587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:19:31.242255 24587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:19:31.247743 24595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:31.247776 24587 server_base.cc:1034] running on GCE node
W20260926 06:19:31.247781 24592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:31.247743 24593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:31.248557 24587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:19:31.248926 24587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:19:31.250105 24587 hybrid_clock.cc:648] HybridClock initialized: now 1790403571250081 us; error 51 us; skew 500 ppm
Sep 26 06:19:31 dist-test-slave-hndd krb5kdc[24574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403571, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:19:31.252844 24587 init.cc:377] Logged in from keytab as kudu/127.2.110.254@KRBTEST.COM (short username kudu)
I20260926 06:19:31.253393 24587 webserver.cc:460] Webserver started at http://127.2.110.254:34143/ using document root <none> and password file <none>
I20260926 06:19:31.253652 24587 fs_manager.cc:359] Metadata directory not provided
I20260926 06:19:31.253726 24587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:19:31.253882 24587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:19:31.255014 24587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790403292369990-2491-0/minicluster-data/master-0/data/instance:
uuid: "034cfd4d08df4c10bf8b638b482da0f6"
format_stamp: "Formatted at 2026-09-26 06:19:31 on dist-test-slave-hndd"
server_key: "970ef0699451875cf638b450c1742c2c"
server_key_iv: "0a400373902baf43b9eb73cbbd040a75"
server_key_version: "encryptionkey@0"
I20260926 06:19:31.255398 24587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790403292369990-2491-0/minicluster-data/master-0/wal/instance:
uuid: "034cfd4d08df4c10bf8b638b482da0f6"
format_stamp: "Formatted at 2026-09-26 06:19:31 on dist-test-slave-hndd"
server_key: "970ef0699451875cf638b450c1742c2c"
server_key_iv: "0a400373902baf43b9eb73cbbd040a75"
server_key_version: "encryptionkey@0"
I20260926 06:19:31.256875 24587 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:19:31.257769 24602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:31.257956 24587 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 06:19:31.258059 24587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "034cfd4d08df4c10bf8b638b482da0f6"
format_stamp: "Formatted at 2026-09-26 06:19:31 on dist-test-slave-hndd"
server_key: "970ef0699451875cf638b450c1742c2c"
server_key_iv: "0a400373902baf43b9eb73cbbd040a75"
server_key_version: "encryptionkey@0"
I20260926 06:19:31.258149 24587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:19:31.261570 24587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:19:31.264113 24587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:19:31.264292 24587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:19:31.268396 24587 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.254:33539
I20260926 06:19:31.268429 24654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.254:33539 every 8 connection(s)
I20260926 06:19:31.268788 24587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:19:31.269865 24655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:19:31.272171 24655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 034cfd4d08df4c10bf8b638b482da0f6: Bootstrap starting.
I20260926 06:19:31.272794 24655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 034cfd4d08df4c10bf8b638b482da0f6: Neither blocks nor log segments found. Creating new log.
I20260926 06:19:31.273084 24655 log.cc:824] T 00000000000000000000000000000000 P 034cfd4d08df4c10bf8b638b482da0f6: Log is configured to *not* fsync() on all Append() calls
I20260926 06:19:31.273873 24655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 034cfd4d08df4c10bf8b638b482da0f6: No bootstrap required, opened a new log
I20260926 06:19:31.275498 24655 raft_consensus.cc:348] T 00000000000000000000000000000000 P 034cfd4d08df4c10bf8b638b482da0f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "034cfd4d08df4c10bf8b638b482da0f6" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 33539 } }
I20260926 06:19:31.275650 24655 raft_consensus.cc:374] T 00000000000000000000000000000000 P 034cfd4d08df4c10bf8b638b482da0f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:19:31.275672 24655 raft_consensus.cc:729] T 00000000000000000000000000000000 P 034cfd4d08df4c10bf8b638b482da0f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 034cfd4d08df4c10bf8b638b482da0f6, State: Initialized, Role: FOLLOWER
I20260926 06:19:31.275842 24655 consensus_queue.cc:260] T 00000000000000000000000000000000 P 034cfd4d08df4c10bf8b638b482da0f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "034cfd4d08df4c10bf8b638b482da0f6" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 33539 } }
I20260926 06:19:31.275908 24655 raft_consensus.cc:388] T 00000000000000000000000000000000 P 034cfd4d08df4c10bf8b638b482da0f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:19:31.275930 24655 raft_consensus.cc:482] T 00000000000000000000000000000000 P 034cfd4d08df4c10bf8b638b482da0f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:19:31.275959 24655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 034cfd4d08df4c10bf8b638b482da0f6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:19:31.276895 24655 raft_consensus.cc:504] T 00000000000000000000000000000000 P 034cfd4d08df4c10bf8b638b482da0f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "034cfd4d08df4c10bf8b638b482da0f6" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 33539 } }
I20260926 06:19:31.276986 24655 leader_election.cc:302] T 00000000000000000000000000000000 P 034cfd4d08df4c10bf8b638b482da0f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 034cfd4d08df4c10bf8b638b482da0f6; no voters:
I20260926 06:19:31.277175 24655 leader_election.cc:288] T 00000000000000000000000000000000 P 034cfd4d08df4c10bf8b638b482da0f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:19:31.277366 24659 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 034cfd4d08df4c10bf8b638b482da0f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:19:31.277478 24655 sys_catalog.cc:564] T 00000000000000000000000000000000 P 034cfd4d08df4c10bf8b638b482da0f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:19:31.277557 24659 raft_consensus.cc:686] T 00000000000000000000000000000000 P 034cfd4d08df4c10bf8b638b482da0f6 [term 1 LEADER]: Becoming Leader. State: Replica: 034cfd4d08df4c10bf8b638b482da0f6, State: Running, Role: LEADER
I20260926 06:19:31.277647 24659 consensus_queue.cc:237] T 00000000000000000000000000000000 P 034cfd4d08df4c10bf8b638b482da0f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "034cfd4d08df4c10bf8b638b482da0f6" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 33539 } }
I20260926 06:19:31.277758 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 24587
I20260926 06:19:31.277885 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790403292369990-2491-0/minicluster-data/master-0/wal/instance
I20260926 06:19:31.278117 2491 external_mini_cluster.cc:1428] Setting key bd24da43be7bad76dc129e7aeb5e0606
I20260926 06:19:31.278331 24658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 034cfd4d08df4c10bf8b638b482da0f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "034cfd4d08df4c10bf8b638b482da0f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "034cfd4d08df4c10bf8b638b482da0f6" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 33539 } } }
I20260926 06:19:31.278358 24660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 034cfd4d08df4c10bf8b638b482da0f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 034cfd4d08df4c10bf8b638b482da0f6. Latest consensus state: current_term: 1 leader_uuid: "034cfd4d08df4c10bf8b638b482da0f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "034cfd4d08df4c10bf8b638b482da0f6" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 33539 } } }
I20260926 06:19:31.278494 24660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 034cfd4d08df4c10bf8b638b482da0f6 [sys.catalog]: This master's current role is: LEADER
I20260926 06:19:31.278713 24658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 034cfd4d08df4c10bf8b638b482da0f6 [sys.catalog]: This master's current role is: LEADER
W20260926 06:19:31.280478 24676 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 034cfd4d08df4c10bf8b638b482da0f6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 06:19:31.280594 24676 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 06:19:31.280755 24671 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:19:31.280936 24671 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:19:31.280964 24675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:31.278762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50719 (local address 127.2.110.254:33539)
0926 06:19:31.279099 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:31.279109 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:31.279147 (+ 38us) server_negotiation.cc:407] Connection header received
0926 06:19:31.279333 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:31.279342 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:31.279376 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:31.279588 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:31.280393 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:31.280397 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:31.280398 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:19:31.280403 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:31.280448 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:19:31.280464 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:31.280543 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:31.280720 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 06:19:31.280767 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":61,"threads_started":1}
I20260926 06:19:31.282001 2491 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:19:31.283435 24671 catalog_manager.cc:1347] Generated new cluster ID: 8a68decb092d42429402aca544c6b7aa
I20260926 06:19:31.283497 24671 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:19:31.284880 24675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:31.283554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36971 (local address 127.2.110.254:33539)
0926 06:19:31.283588 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:31.283590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:31.283875 (+ 285us) server_negotiation.cc:407] Connection header received
0926 06:19:31.283964 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:31.283965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:31.283970 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:31.284019 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:31.284639 (+ 620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:31.284641 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:31.284641 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:19:31.284645 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:31.284667 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:19:31.284672 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:31.284728 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:31.284806 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 06:19:31.284820 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 06:19:31.293452 24671 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:19:31.293718 24671 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:19:31.298750 24671 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 034cfd4d08df4c10bf8b638b482da0f6: Generated new TSK 0
I20260926 06:19:31.298971 24671 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:19:31.307437 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 24587
2026-09-26T06:19:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1341 ms)
[----------] 18 tests from AuthTokenIssuingTest (24523 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:19:31 dist-test-slave-hndd krb5kdc[24686](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:19:31 dist-test-slave-hndd krb5kdc[24686](info): set up 2 sockets
Sep 26 06:19:31 dist-test-slave-hndd krb5kdc[24686](info): commencing operation
krb5kdc: starting...
W20260926 06:19:32.361809 2491 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Sep 26 06:19:32 dist-test-slave-hndd krb5kdc[24686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:19:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:19:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
I20260926 06:19:32.485278 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:33011
--webserver_interface=127.2.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:42909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.110.254:33011,127.2.110.253:45513,127.2.110.252:44051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:19:32.569211 24699 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 06:19:32.569533 24699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:19:32.569572 24699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:19:32.569615 24699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:19:32.570806 24699 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:19:32.570876 24699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:19:32.570901 24699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:19:32.570923 24699 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:19:32.570950 24699 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:19:32.572499 24699 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:42909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.110.254:33011,127.2.110.253:45513,127.2.110.252:44051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:33011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:19:32.572798 24699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:19:32.573037 24699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:19:32.578285 24707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:32.578400 24705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:32.578363 24704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:32.578840 24699 server_base.cc:1034] running on GCE node
I20260926 06:19:32.579181 24699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:19:32.579509 24699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:19:32.580683 24699 hybrid_clock.cc:648] HybridClock initialized: now 1790403572580678 us; error 40 us; skew 500 ppm
Sep 26 06:19:32 dist-test-slave-hndd krb5kdc[24686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403572, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:19:32.584172 24699 init.cc:377] Logged in from keytab as kudu/127.2.110.254@KRBTEST.COM (short username kudu)
I20260926 06:19:32.584895 24699 webserver.cc:460] Webserver started at http://127.2.110.254:33135/ using document root <none> and password file <none>
I20260926 06:19:32.585225 24699 fs_manager.cc:359] Metadata directory not provided
I20260926 06:19:32.585315 24699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:19:32.585687 24699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:19:32.586923 24699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/minicluster-data/master-0/data/instance:
uuid: "724e9816f07f4c0abb83624e37850698"
format_stamp: "Formatted at 2026-09-26 06:19:32 on dist-test-slave-hndd"
server_key: "0a2bb976062ffa89f628518bc36325df"
server_key_iv: "4b85f0b27b3869275194af2cee5f4a78"
server_key_version: "encryptionkey@0"
I20260926 06:19:32.587327 24699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/minicluster-data/master-0/wal/instance:
uuid: "724e9816f07f4c0abb83624e37850698"
format_stamp: "Formatted at 2026-09-26 06:19:32 on dist-test-slave-hndd"
server_key: "0a2bb976062ffa89f628518bc36325df"
server_key_iv: "4b85f0b27b3869275194af2cee5f4a78"
server_key_version: "encryptionkey@0"
I20260926 06:19:32.588745 24699 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:19:32.589651 24714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:32.589833 24699 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:19:32.589946 24699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "724e9816f07f4c0abb83624e37850698"
format_stamp: "Formatted at 2026-09-26 06:19:32 on dist-test-slave-hndd"
server_key: "0a2bb976062ffa89f628518bc36325df"
server_key_iv: "4b85f0b27b3869275194af2cee5f4a78"
server_key_version: "encryptionkey@0"
I20260926 06:19:32.590035 24699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:19:32.600023 24699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:19:32.602041 24699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:19:32.602216 24699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:19:32.606652 24699 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.254:33011
I20260926 06:19:32.606700 24766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.254:33011 every 8 connection(s)
I20260926 06:19:32.607075 24699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:19:32.608256 24767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:19:32.610069 24767 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.110.254" port: 33011 } has no permanent_uuid. Determining permanent_uuid...
I20260926 06:19:32.611236 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 24699
I20260926 06:19:32.611354 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/minicluster-data/master-0/wal/instance
I20260926 06:19:32.611548 2491 external_mini_cluster.cc:1428] Setting key 2001935c2c05d0a3dc027ba1e9490ff5
Sep 26 06:19:32 dist-test-slave-hndd krb5kdc[24686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403572, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.254@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.110.253@KRBTEST.COM; defaulting to no policy
I20260926 06:19:32.621691 24769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:32.610834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:33011 (local address 127.0.0.1:59168)
0926 06:19:32.611176 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:32.611240 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:32.611895 (+ 655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:32.612234 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:19:32.612237 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:19:32.612500 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:19:32.612663 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:32.612667 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:32.613825 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:32.613826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:32.614991 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:32.614995 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:32.615585 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:32.615587 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:32.615669 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:32.616353 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:19:32.616365 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:19:32.618144 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:19:32.620026 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:19:32.620033 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:19:32.620035 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:19:32.620230 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:19:32.620512 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:19:32.620513 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:19:32.620513 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:19:32.620600 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:19:32.621072 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:19:32.621078 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:19:32.621140 (+ 62us) client_negotiation.cc:769] Sending connection context
0926 06:19:32.621273 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 06:19:32.621489 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":195,"threads_started":1}
I20260926 06:19:32.621695 24770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:32.610916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59168 (local address 127.2.110.254:33011)
0926 06:19:32.611244 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:32.611255 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:32.611518 (+ 263us) server_negotiation.cc:407] Connection header received
0926 06:19:32.612072 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:32.612075 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:32.612098 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:32.612207 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:32.612762 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:32.613737 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:32.615090 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:32.615467 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:32.615554 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:32.618253 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:32.618260 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:32.618260 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:19:32.618272 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:32.619923 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:32.620323 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:32.620327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:32.620327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:32.620438 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:32.620715 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:32.620716 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:32.620716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:32.620965 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:32.621063 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:32.621385 (+ 322us) server_negotiation.cc:299] Negotiation successful
0926 06:19:32.621539 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":68,"server-negotiator.queue_time_us":287,"thread_start_us":175,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.253@KRBTEST.COM" created.
I20260926 06:19:32.622459 24767 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.110.253" port: 45513 } has no permanent_uuid. Determining permanent_uuid...
I20260926 06:19:32.622836 24769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:32.622720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.253:45513 (local address 127.0.0.1:34908)
0926 06:19:32.622775 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:32.622817 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.253:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 06:19:32.622967 24717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.110.253:45513: connect: Connection refused (error 111)
I20260926 06:19:32.623395 24769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:32.623303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.253:45513 (local address 127.0.0.1:34914)
0926 06:19:32.623350 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:32.623374 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.253:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 06:19:32.623517 24767 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.110.253:45513: Network error: Client connection negotiation failed: client connection to 127.2.110.253:45513: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.253.keytab.
Entry for principal kudu/127.2.110.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.253.keytab.
I20260926 06:19:32.647816 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.253:45513
--webserver_interface=127.2.110.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.253.keytab
--principal=kudu/127.2.110.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:42909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.110.254:33011,127.2.110.253:45513,127.2.110.252:44051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 06:19:32.672785 24767 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.110.253" port: 45513 } attempt: 1
I20260926 06:19:32.673296 24769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:32.673169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.253:45513 (local address 127.0.0.1:34918)
0926 06:19:32.673229 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:32.673267 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.253:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 06:19:32.673650 24769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:32.673555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.253:45513 (local address 127.0.0.1:34928)
0926 06:19:32.673597 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:32.673622 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.253:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 06:19:32.673732 24767 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.110.253:45513: Network error: Client connection negotiation failed: client connection to 127.2.110.253:45513: connect: Connection refused (error 111)
W20260926 06:19:32.732861 24774 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 06:19:32.733271 24774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:19:32.733357 24774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:19:32.733409 24774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:19:32.734807 24774 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:19:32.734894 24774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:19:32.734923 24774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:19:32.734957 24774 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:19:32.735064 24774 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:19:32.736949 24774 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:42909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.110.254:33011,127.2.110.253:45513,127.2.110.252:44051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.253.keytab
--principal=kudu/127.2.110.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.253:45513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.110.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:19:32.737275 24774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:19:32.737552 24774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 06:19:32.741859 24767 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.110.253" port: 45513 } attempt: 2
I20260926 06:19:32.742327 24769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:32.742197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.253:45513 (local address 127.0.0.1:34944)
0926 06:19:32.742257 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:32.742289 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.253:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 06:19:32.742579 24780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:32.742558 24779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:32.742673 24769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:32.742570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.253:45513 (local address 127.0.0.1:34960)
0926 06:19:32.742607 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:32.742630 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.253:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260926 06:19:32.742550 24782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:32.742834 24767 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.110.253:45513: Network error: Client connection negotiation failed: client connection to 127.2.110.253:45513: connect: Connection refused (error 111)
I20260926 06:19:32.742980 24774 server_base.cc:1034] running on GCE node
I20260926 06:19:32.743286 24774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:19:32.743548 24774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:19:32.744724 24774 hybrid_clock.cc:648] HybridClock initialized: now 1790403572744696 us; error 40 us; skew 500 ppm
Sep 26 06:19:32 dist-test-slave-hndd krb5kdc[24686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403572, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:19:32.747345 24774 init.cc:377] Logged in from keytab as kudu/127.2.110.253@KRBTEST.COM (short username kudu)
I20260926 06:19:32.747767 24774 webserver.cc:460] Webserver started at http://127.2.110.253:39889/ using document root <none> and password file <none>
I20260926 06:19:32.748029 24774 fs_manager.cc:359] Metadata directory not provided
I20260926 06:19:32.748099 24774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:19:32.748242 24774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:19:32.749388 24774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/minicluster-data/master-1/data/instance:
uuid: "148e04781dea445999e20ed86b870db8"
format_stamp: "Formatted at 2026-09-26 06:19:32 on dist-test-slave-hndd"
server_key: "58b33efbc3d72a3a026b6bf2b4d612cb"
server_key_iv: "1d944c64cbd7c79dcfdce70d4f7c771b"
server_key_version: "encryptionkey@0"
I20260926 06:19:32.749768 24774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/minicluster-data/master-1/wal/instance:
uuid: "148e04781dea445999e20ed86b870db8"
format_stamp: "Formatted at 2026-09-26 06:19:32 on dist-test-slave-hndd"
server_key: "58b33efbc3d72a3a026b6bf2b4d612cb"
server_key_iv: "1d944c64cbd7c79dcfdce70d4f7c771b"
server_key_version: "encryptionkey@0"
I20260926 06:19:32.751151 24774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:19:32.751943 24789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:32.752137 24774 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:19:32.752252 24774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/minicluster-data/master-1/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/minicluster-data/master-1/wal
uuid: "148e04781dea445999e20ed86b870db8"
format_stamp: "Formatted at 2026-09-26 06:19:32 on dist-test-slave-hndd"
server_key: "58b33efbc3d72a3a026b6bf2b4d612cb"
server_key_iv: "1d944c64cbd7c79dcfdce70d4f7c771b"
server_key_version: "encryptionkey@0"
I20260926 06:19:32.752333 24774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:19:32.763965 24774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:19:32.766218 24774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:19:32.766392 24774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:19:32.770500 24774 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.253:45513
I20260926 06:19:32.770543 24841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.253:45513 every 8 connection(s)
I20260926 06:19:32.770924 24774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/minicluster-data/master-1/data/info.pb
I20260926 06:19:32.771917 24842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:19:32.773329 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 24774
I20260926 06:19:32.773486 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/minicluster-data/master-1/wal/instance
I20260926 06:19:32.773674 2491 external_mini_cluster.cc:1428] Setting key 729914d1e9fd0010284141d89efc38e1
I20260926 06:19:32.773612 24842 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.110.254" port: 33011 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 06:19:32 dist-test-slave-hndd krb5kdc[24686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403572, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.253@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.110.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.252@KRBTEST.COM" created.
I20260926 06:19:32.783708 24770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:32.774433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59178 (local address 127.2.110.254:33011)
0926 06:19:32.774496 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:32.774498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:32.774764 (+ 266us) server_negotiation.cc:407] Connection header received
0926 06:19:32.775278 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:32.775279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:32.775294 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:32.775353 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:32.775998 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:32.776517 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:32.777646 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:32.777861 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:32.777936 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:32.780387 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:32.780388 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:32.780389 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:19:32.780398 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:32.782006 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:32.782511 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:32.782512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:32.782513 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:32.782578 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:32.782934 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:32.782935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:32.782935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:32.783077 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:32.783152 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:32.783498 (+ 346us) server_negotiation.cc:299] Negotiation successful
0926 06:19:32.783599 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 06:19:32.783933 24844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:32.774416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:33011 (local address 127.0.0.1:59178)
0926 06:19:32.774639 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:32.774676 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:32.775173 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:32.775422 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:19:32.775426 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:19:32.775692 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:19:32.775906 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:32.775910 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:32.776615 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:32.776616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:32.777530 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:32.777533 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:32.777958 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:32.777959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:32.778090 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:32.778730 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:19:32.778739 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:19:32.780261 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:19:32.782143 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:19:32.782152 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:19:32.782154 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:19:32.782412 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:19:32.782651 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:19:32.782652 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:19:32.782653 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:19:32.782848 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:19:32.783156 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:19:32.783164 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:19:32.783354 (+ 190us) client_negotiation.cc:769] Sending connection context
0926 06:19:32.783476 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 06:19:32.783665 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":85,"threads_started":1}
I20260926 06:19:32.784559 24842 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.110.253" port: 45513 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 06:19:32 dist-test-slave-hndd krb5kdc[24686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403572, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.253@KRBTEST.COM for kudu/127.2.110.253@KRBTEST.COM
I20260926 06:19:32.793025 24844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:32.784901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.253:45513 (local address 127.0.0.1:34968)
0926 06:19:32.784951 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:32.784963 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:32.785005 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:32.785272 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:19:32.785273 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:19:32.785456 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:19:32.785508 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:32.785510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:32.786510 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:32.786512 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:32.787077 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:32.787079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:32.787621 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:32.787622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:32.787709 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:32.788217 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:19:32.788221 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:19:32.789566 (+ 1345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:19:32.791771 (+ 2205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:19:32.791772 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:19:32.791772 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:19:32.791949 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:19:32.792140 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:19:32.792141 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:19:32.792141 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:19:32.792227 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:19:32.792596 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:19:32.792597 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:19:32.792641 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 06:19:32.792732 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 06:19:32.792892 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 06:19:32.793113 24847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:32.784953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34968 (local address 127.2.110.253:45513)
0926 06:19:32.785138 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:32.785142 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:32.785155 (+ 13us) server_negotiation.cc:407] Connection header received
0926 06:19:32.785174 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:32.785175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:32.785187 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:32.785272 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:32.785596 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:32.786396 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:32.787172 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:32.787527 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:32.787623 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:32.789663 (+ 2040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:32.789674 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:32.789675 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:19:32.789683 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:32.791676 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:32.792023 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:32.792026 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:32.792026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:32.792082 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:32.792307 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:32.792308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:32.792308 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:32.792516 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:32.792610 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:32.792768 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 06:19:32.792957 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":154,"thread_start_us":98,"threads_started":1}
I20260926 06:19:32.793825 24842 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.110.252" port: 44051 } has no permanent_uuid. Determining permanent_uuid...
I20260926 06:19:32.794256 24844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:32.794126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.252:44051 (local address 127.0.0.1:57622)
0926 06:19:32.794177 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:32.794228 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.252:44051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 06:19:32.794395 24792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.110.252:44051: connect: Connection refused (error 111)
I20260926 06:19:32.794813 24844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:32.794713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.252:44051 (local address 127.0.0.1:57624)
0926 06:19:32.794764 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:32.794790 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.252:44051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 06:19:32.794924 24842 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.110.252:44051: Network error: Client connection negotiation failed: client connection to 127.2.110.252:44051: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.252.keytab.
Entry for principal kudu/127.2.110.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.252.keytab.
I20260926 06:19:32.798431 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.252:44051
--webserver_interface=127.2.110.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.252.keytab
--principal=kudu/127.2.110.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:42909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.110.254:33011,127.2.110.253:45513,127.2.110.252:44051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 06:19:32.834105 24767 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.110.253" port: 45513 } attempt: 3
Sep 26 06:19:32 dist-test-slave-hndd krb5kdc[24686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403572, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.254@KRBTEST.COM for kudu/127.2.110.253@KRBTEST.COM
I20260926 06:19:32.841998 24847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:32.834653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34976 (local address 127.2.110.253:45513)
0926 06:19:32.834728 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:32.834730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:32.834741 (+ 11us) server_negotiation.cc:407] Connection header received
0926 06:19:32.834796 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:32.834797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:32.834816 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:32.834889 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:32.835242 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:32.835788 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:32.836577 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:32.836836 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:32.836923 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:32.839227 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:32.839228 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:32.839228 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:19:32.839236 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:32.840793 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:32.841146 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:32.841147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:32.841147 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:32.841205 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:32.841444 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:32.841444 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:32.841445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:32.841572 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:32.841645 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:32.841810 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 06:19:32.841896 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 06:19:32.841998 24769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:32.834579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.253:45513 (local address 127.0.0.1:34976)
0926 06:19:32.834649 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:32.834660 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:32.834725 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:32.834894 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:19:32.834895 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:19:32.835084 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:19:32.835152 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:32.835154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:32.835889 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:32.835891 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:32.836471 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:32.836473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:32.836920 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:32.836921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:32.837012 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:32.837641 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:19:32.837646 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:19:32.839123 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:19:32.840884 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:19:32.840886 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:19:32.840886 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:19:32.841074 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:19:32.841281 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:19:32.841282 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:19:32.841282 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:19:32.841387 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:19:32.841647 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:19:32.841648 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:19:32.841699 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 06:19:32.841785 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 06:19:32.841894 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 06:19:32.842464 24767 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.110.252" port: 44051 } has no permanent_uuid. Determining permanent_uuid...
I20260926 06:19:32.842808 24769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:32.842693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.252:44051 (local address 127.0.0.1:57634)
0926 06:19:32.842743 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:32.842781 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.252:44051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 06:19:32.843080 24769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:32.843007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.252:44051 (local address 127.0.0.1:57636)
0926 06:19:32.843038 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:32.843061 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.252:44051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260926 06:19:32.843283 24767 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.110.252:44051: Network error: Client connection negotiation failed: client connection to 127.2.110.252:44051: connect: Connection refused (error 111)
I20260926 06:19:32.844154 24842 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.110.252" port: 44051 } attempt: 1
I20260926 06:19:32.844508 24844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:32.844403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.252:44051 (local address 127.0.0.1:57640)
0926 06:19:32.844453 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:32.844484 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.252:44051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 06:19:32.844843 24844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:32.844789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.252:44051 (local address 127.0.0.1:57654)
0926 06:19:32.844811 (+ 22us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:32.844831 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.252:44051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260926 06:19:32.844933 24842 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.110.252:44051: Network error: Client connection negotiation failed: client connection to 127.2.110.252:44051: connect: Connection refused (error 111)
I20260926 06:19:32.874435 24767 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.110.252" port: 44051 } attempt: 1
I20260926 06:19:32.874965 24769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:32.874800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.252:44051 (local address 127.0.0.1:57666)
0926 06:19:32.874877 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:32.874924 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.252:44051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 06:19:32.875311 24769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:32.875222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.252:44051 (local address 127.0.0.1:57672)
0926 06:19:32.875259 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:32.875288 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.252:44051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260926 06:19:32.875412 24767 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.110.252:44051: Network error: Client connection negotiation failed: client connection to 127.2.110.252:44051: connect: Connection refused (error 111)
W20260926 06:19:32.886979 24849 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 06:19:32.887205 24849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:19:32.887223 24849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:19:32.887243 24849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:19:32.888417 24849 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:19:32.888464 24849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:19:32.888478 24849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:19:32.888491 24849 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:19:32.888504 24849 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:19:32.890120 24849 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:42909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.110.254:33011,127.2.110.253:45513,127.2.110.252:44051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.252.keytab
--principal=kudu/127.2.110.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.252:44051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.110.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:19:32.890388 24849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:19:32.890626 24849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:19:32.895943 24855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:32.895932 24857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:32.896009 24849 server_base.cc:1034] running on GCE node
W20260926 06:19:32.895946 24854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:32.896391 24849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:19:32.896723 24849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:19:32.897913 24849 hybrid_clock.cc:648] HybridClock initialized: now 1790403572897885 us; error 49 us; skew 500 ppm
Sep 26 06:19:32 dist-test-slave-hndd krb5kdc[24686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403572, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:19:32.900645 24849 init.cc:377] Logged in from keytab as kudu/127.2.110.252@KRBTEST.COM (short username kudu)
I20260926 06:19:32.901121 24849 webserver.cc:460] Webserver started at http://127.2.110.252:42293/ using document root <none> and password file <none>
I20260926 06:19:32.901381 24849 fs_manager.cc:359] Metadata directory not provided
I20260926 06:19:32.901446 24849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:19:32.901562 24849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:19:32.902696 24849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/minicluster-data/master-2/data/instance:
uuid: "3cc49582c3a34406b0ca4ab50ac6fc63"
format_stamp: "Formatted at 2026-09-26 06:19:32 on dist-test-slave-hndd"
server_key: "176e04599f4ec9e47fd45a0598dd7b22"
server_key_iv: "c010744087232e60b55232be3ce7bdc6"
server_key_version: "encryptionkey@0"
I20260926 06:19:32.903038 24849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/minicluster-data/master-2/wal/instance:
uuid: "3cc49582c3a34406b0ca4ab50ac6fc63"
format_stamp: "Formatted at 2026-09-26 06:19:32 on dist-test-slave-hndd"
server_key: "176e04599f4ec9e47fd45a0598dd7b22"
server_key_iv: "c010744087232e60b55232be3ce7bdc6"
server_key_version: "encryptionkey@0"
I20260926 06:19:32.904465 24849 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:19:32.905298 24864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:32.905493 24849 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:19:32.905581 24849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/minicluster-data/master-2/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/minicluster-data/master-2/wal
uuid: "3cc49582c3a34406b0ca4ab50ac6fc63"
format_stamp: "Formatted at 2026-09-26 06:19:32 on dist-test-slave-hndd"
server_key: "176e04599f4ec9e47fd45a0598dd7b22"
server_key_iv: "c010744087232e60b55232be3ce7bdc6"
server_key_version: "encryptionkey@0"
I20260926 06:19:32.905640 24849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:19:32.913100 24842 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.110.252" port: 44051 } attempt: 2
I20260926 06:19:32.913725 24844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:32.913592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.252:44051 (local address 127.0.0.1:57684)
0926 06:19:32.913654 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:32.913695 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.252:44051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 06:19:32.914104 24844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:32.914022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.252:44051 (local address 127.0.0.1:57690)
0926 06:19:32.914063 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:32.914088 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.252:44051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 06:19:32.914206 24842 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.110.252:44051: Network error: Client connection negotiation failed: client connection to 127.2.110.252:44051: connect: Connection refused (error 111)
I20260926 06:19:32.918759 24849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:19:32.921073 24849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:19:32.921239 24849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:19:32.925256 24849 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.252:44051
I20260926 06:19:32.925300 24916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.252:44051 every 8 connection(s)
I20260926 06:19:32.925715 24849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/minicluster-data/master-2/data/info.pb
I20260926 06:19:32.926874 24917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:19:32.928505 24917 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.110.254" port: 33011 } has no permanent_uuid. Determining permanent_uuid...
I20260926 06:19:32.933955 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 24849
Sep 26 06:19:32 dist-test-slave-hndd krb5kdc[24686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403572, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.252@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:19:32.934098 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403292369990-2491-0/minicluster-data/master-2/wal/instance
I20260926 06:19:32.934314 2491 external_mini_cluster.cc:1428] Setting key 3d442e73b564e3ce55fe702fb2f75108
I20260926 06:19:32.937991 24770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:32.929267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59180 (local address 127.2.110.254:33011)
0926 06:19:32.929394 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:32.929396 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:32.929525 (+ 129us) server_negotiation.cc:407] Connection header received
0926 06:19:32.929981 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:32.929982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:32.929992 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:32.930040 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:32.930614 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:32.931089 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:32.932086 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:32.932282 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:32.932350 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:32.934780 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:32.934782 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:32.934782 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:19:32.934792 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:32.936438 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:32.936893 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:32.936894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:32.936894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:32.936955 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:32.937223 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:32.937224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:32.937224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:32.937330 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:32.937430 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:32.937802 (+ 372us) server_negotiation.cc:299] Negotiation successful
0926 06:19:32.937897 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260926 06:19:32.938069 24919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:32.929191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:33011 (local address 127.0.0.1:59180)
0926 06:19:32.929395 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:32.929443 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:32.929898 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:32.930103 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:19:32.930106 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:19:32.930350 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:19:32.930546 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:32.930549 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:32.931203 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:32.931204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:32.931990 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:32.931993 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:32.932378 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:32.932378 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:32.932486 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:32.932975 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:19:32.932984 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:19:32.934615 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:19:32.936526 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:19:32.936533 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:19:32.936536 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:19:32.936815 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:19:32.937015 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:19:32.937016 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:19:32.937016 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:19:32.937156 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:19:32.937424 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:19:32.937430 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:19:32.937589 (+ 159us) client_negotiation.cc:769] Sending connection context
0926 06:19:32.937720 (+ 131us) client_negotiation.cc:240] Negotiation successful
0926 06:19:32.937877 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":77,"threads_started":1}
I20260926 06:19:32.938678 24917 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.110.253" port: 45513 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 06:19:32 dist-test-slave-hndd krb5kdc[24686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.110.252@KRBTEST.COM
Sep 26 06:19:32 dist-test-slave-hndd krb5kdc[24686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403572, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.252@KRBTEST.COM for kudu/127.2.110.253@KRBTEST.COM
I20260926 06:19:32.944380 24921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:32.934718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57706 (local address 127.2.110.252:44051)
0926 06:19:32.934888 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:32.934892 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:32.934977 (+ 85us) server_negotiation.cc:407] Connection header received
0926 06:19:32.935080 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:32.935086 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:32.935099 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:32.935162 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:32.935990 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:32.936706 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:32.937772 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:32.938242 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:32.938345 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:32.940995 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:32.941004 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:32.941005 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:19:32.941012 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:32.942875 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:32.943334 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:32.943338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:32.943338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:32.943410 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:32.943637 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:32.943638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:32.943638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:32.943862 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:32.944026 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:32.944132 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 06:19:32.944235 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":92,"threads_started":1}
I20260926 06:19:32.945933 24847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:32.939047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34988 (local address 127.2.110.253:45513)
0926 06:19:32.939114 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:32.939116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:32.939124 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:19:32.939146 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:32.939147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:32.939156 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:32.939233 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:32.939511 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:32.939961 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:32.940637 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:32.940854 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:32.940924 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:32.942705 (+ 1781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:32.942706 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:32.942707 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:19:32.942716 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:32.944408 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:32.944805 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:32.944806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:32.944806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:32.944870 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:32.945290 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:32.945291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:32.945292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:32.945484 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:32.945570 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:32.945798 (+ 228us) server_negotiation.cc:299] Negotiation successful
0926 06:19:32.945873 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 06:19:32.945986 24919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:32.938961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.253:45513 (local address 127.0.0.1:34988)
0926 06:19:32.938998 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:32.939009 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:32.939055 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:32.939238 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:19:32.939239 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:19:32.939380 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:19:32.939430 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:32.939431 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:32.940047 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:32.940048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:32.940554 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:32.940556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:32.940933 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:32.940934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:32.941008 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:32.941407 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:19:32.941413 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:19:32.942622 (+ 1209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:19:32.944514 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:19:32.944515 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:19:32.944515 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:19:32.944734 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:19:32.945047 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:19:32.945048 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:19:32.945049 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:19:32.945126 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:19:32.945604 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:19:32.945605 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:19:32.945656 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 06:19:32.945739 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 06:19:32.945845 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260926 06:19:32.946465 24917 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.110.252" port: 44051 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 06:19:32 dist-test-slave-hndd krb5kdc[24686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:19:32.950591 24767 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.110.252" port: 44051 } attempt: 2
Sep 26 06:19:32 dist-test-slave-hndd krb5kdc[24686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403572, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.252@KRBTEST.COM for kudu/127.2.110.252@KRBTEST.COM
I20260926 06:19:32.954576 24919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:32.946724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.252:44051 (local address 127.0.0.1:57714)
0926 06:19:32.946772 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:32.946782 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:32.946817 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:32.946941 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:19:32.946941 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:19:32.947050 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:19:32.947090 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:32.947091 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:32.947812 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:32.947813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:32.948477 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:32.948479 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:32.949017 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:32.949018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:32.949089 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:32.949527 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:19:32.949533 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:19:32.951506 (+ 1973us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:19:32.953547 (+ 2041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:19:32.953548 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:19:32.953548 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:19:32.953768 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:19:32.953967 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:19:32.953968 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:19:32.953968 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:19:32.954021 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:19:32.954278 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:19:32.954278 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:19:32.954316 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 06:19:32.954380 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 06:19:32.954477 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 06:19:32.954576 24770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:32.945500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59190 (local address 127.2.110.254:33011)
0926 06:19:32.945542 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:32.945544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:32.945561 (+ 17us) server_negotiation.cc:407] Connection header received
0926 06:19:32.945646 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:32.945647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:32.945655 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:32.945699 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:32.946351 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:32.947114 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:32.947914 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:32.948157 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:32.948212 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:32.951092 (+ 2880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:32.951093 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:32.951093 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:19:32.951101 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:32.953033 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:32.953469 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:32.953470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:32.953471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:32.953538 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:32.953905 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:32.953906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:32.953906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:32.954040 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:32.954193 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:32.954394 (+ 201us) server_negotiation.cc:299] Negotiation successful
0926 06:19:32.954477 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 06:19:32.955029 24921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:32.946789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57714 (local address 127.2.110.252:44051)
0926 06:19:32.946826 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:32.946828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:32.946842 (+ 14us) server_negotiation.cc:407] Connection header received
0926 06:19:32.946862 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:32.946862 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:32.946871 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:32.946939 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:32.947159 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:32.947686 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:32.948575 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:32.948911 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:32.948972 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:32.951580 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:32.951581 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:32.951581 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:19:32.951589 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:32.953473 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:32.953846 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:32.953847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:32.953848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:32.953909 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:32.954089 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:32.954090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:32.954091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:32.954212 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:32.954285 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:32.954868 (+ 583us) server_negotiation.cc:299] Negotiation successful
0926 06:19:32.954952 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 06:19:32.956735 24917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cc49582c3a34406b0ca4ab50ac6fc63: Bootstrap starting.
Sep 26 06:19:32 dist-test-slave-hndd krb5kdc[24686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403572, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.254@KRBTEST.COM for kudu/127.2.110.252@KRBTEST.COM
I20260926 06:19:32.957458 24917 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3cc49582c3a34406b0ca4ab50ac6fc63: Neither blocks nor log segments found. Creating new log.
I20260926 06:19:32.957809 24917 log.cc:824] T 00000000000000000000000000000000 P 3cc49582c3a34406b0ca4ab50ac6fc63: Log is configured to *not* fsync() on all Append() calls
I20260926 06:19:32.958680 24917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cc49582c3a34406b0ca4ab50ac6fc63: No bootstrap required, opened a new log
I20260926 06:19:32.960615 24917 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3cc49582c3a34406b0ca4ab50ac6fc63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "724e9816f07f4c0abb83624e37850698" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 33011 } } peers { permanent_uuid: "148e04781dea445999e20ed86b870db8" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 45513 } } peers { permanent_uuid: "3cc49582c3a34406b0ca4ab50ac6fc63" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 44051 } }
I20260926 06:19:32.960781 24917 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3cc49582c3a34406b0ca4ab50ac6fc63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cc49582c3a34406b0ca4ab50ac6fc63, State: Initialized, Role: FOLLOWER
I20260926 06:19:32.960867 24917 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3cc49582c3a34406b0ca4ab50ac6fc63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "724e9816f07f4c0abb83624e37850698" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 33011 } } peers { permanent_uuid: "148e04781dea445999e20ed86b870db8" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 45513 } } peers { permanent_uuid: "3cc49582c3a34406b0ca4ab50ac6fc63" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 44051 } }
I20260926 06:19:32.961185 24924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cc49582c3a34406b0ca4ab50ac6fc63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "724e9816f07f4c0abb83624e37850698" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 33011 } } peers { permanent_uuid: "148e04781dea445999e20ed86b870db8" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 45513 } } peers { permanent_uuid: "3cc49582c3a34406b0ca4ab50ac6fc63" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 44051 } } }
I20260926 06:19:32.961220 24917 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3cc49582c3a34406b0ca4ab50ac6fc63 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:19:32.962646 24924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cc49582c3a34406b0ca4ab50ac6fc63 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 06:19:32.963337 24935 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3cc49582c3a34406b0ca4ab50ac6fc63: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 06:19:32.963389 24935 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 26 06:19:32 dist-test-slave-hndd krb5kdc[24686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.110.253@KRBTEST.COM
I20260926 06:19:32.963507 24769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:32.950878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.252:44051 (local address 127.0.0.1:57728)
0926 06:19:32.951417 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:32.951430 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:32.951476 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:32.951840 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:19:32.951842 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:19:32.951983 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:19:32.952029 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:32.952031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:32.953429 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:32.953430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:32.954237 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:32.954240 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:32.955419 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:32.955420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:32.955485 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:32.955867 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:19:32.955871 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:19:32.957137 (+ 1266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:19:32.962203 (+ 5066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:19:32.962204 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:19:32.962205 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:19:32.962498 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:19:32.962776 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:19:32.962777 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:19:32.962777 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:19:32.962851 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:19:32.963156 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:19:32.963157 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:19:32.963221 (+ 64us) client_negotiation.cc:769] Sending connection context
0926 06:19:32.963287 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 06:19:32.963404 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519}
I20260926 06:19:32.963723 24922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:32.951276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57728 (local address 127.2.110.252:44051)
0926 06:19:32.951692 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:32.951694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:32.951703 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:19:32.951724 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:32.951725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:32.951737 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:32.952441 (+ 704us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:32.952504 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:32.953309 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:32.954325 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:32.955313 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:32.955418 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:32.957225 (+ 1807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:32.957226 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:32.957226 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:19:32.957234 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:32.962069 (+ 4835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:32.962599 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:32.962600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:32.962601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:32.962679 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:32.962922 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:32.962923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:32.962923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:32.963079 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:32.963163 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:32.963336 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 06:19:32.963426 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":98,"threads_started":1}
I20260926 06:19:32.965936 24767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 724e9816f07f4c0abb83624e37850698: Bootstrap starting.
I20260926 06:19:32.964258 24849 master_runner.cc:186] Error getting master registration for 127.2.110.254:33011: OK, instance_id { permanent_uuid: "724e9816f07f4c0abb83624e37850698" instance_seqno: 1790403572605151 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 06:19:32.966753 24767 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 724e9816f07f4c0abb83624e37850698: Neither blocks nor log segments found. Creating new log.
I20260926 06:19:32.966842 24849 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 06:19:32.967106 24767 log.cc:824] T 00000000000000000000000000000000 P 724e9816f07f4c0abb83624e37850698: Log is configured to *not* fsync() on all Append() calls
I20260926 06:19:32.967867 24847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:32.958499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35004 (local address 127.2.110.253:45513)
0926 06:19:32.958558 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:32.958560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:32.958568 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:19:32.958589 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:32.958590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:32.958599 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:32.958647 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:32.959242 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:32.959717 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:32.960701 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:32.961000 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:32.961076 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:32.964463 (+ 3387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:32.964464 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:32.964465 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:19:32.964473 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:32.966361 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:32.966938 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:32.966940 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:32.966940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:32.967011 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:32.967285 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:32.967286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:32.967286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:32.967442 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:32.967605 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:32.967660 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 06:19:32.967743 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 06:19:32.967976 24767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 724e9816f07f4c0abb83624e37850698: No bootstrap required, opened a new log
I20260926 06:19:32.970227 24767 raft_consensus.cc:348] T 00000000000000000000000000000000 P 724e9816f07f4c0abb83624e37850698 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "724e9816f07f4c0abb83624e37850698" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 33011 } } peers { permanent_uuid: "148e04781dea445999e20ed86b870db8" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 45513 } } peers { permanent_uuid: "3cc49582c3a34406b0ca4ab50ac6fc63" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 44051 } }
I20260926 06:19:32.970402 24767 raft_consensus.cc:729] T 00000000000000000000000000000000 P 724e9816f07f4c0abb83624e37850698 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 724e9816f07f4c0abb83624e37850698, State: Initialized, Role: FOLLOWER
I20260926 06:19:32.970503 24767 consensus_queue.cc:260] T 00000000000000000000000000000000 P 724e9816f07f4c0abb83624e37850698 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "724e9816f07f4c0abb83624e37850698" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 33011 } } peers { permanent_uuid: "148e04781dea445999e20ed86b870db8" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 45513 } } peers { permanent_uuid: "3cc49582c3a34406b0ca4ab50ac6fc63" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 44051 } }
I20260926 06:19:32.970641 2491 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:19:32.970892 24767 sys_catalog.cc:564] T 00000000000000000000000000000000 P 724e9816f07f4c0abb83624e37850698 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:19:32.971553 24741 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "724e9816f07f4c0abb83624e37850698"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59190
I20260926 06:19:32.971688 24741 raft_consensus.cc:482] T 00000000000000000000000000000000 P 724e9816f07f4c0abb83624e37850698 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 06:19:32.971747 24741 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 724e9816f07f4c0abb83624e37850698 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:19:32.972697 24741 raft_consensus.cc:504] T 00000000000000000000000000000000 P 724e9816f07f4c0abb83624e37850698 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "724e9816f07f4c0abb83624e37850698" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 33011 } } peers { permanent_uuid: "148e04781dea445999e20ed86b870db8" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 45513 } } peers { permanent_uuid: "3cc49582c3a34406b0ca4ab50ac6fc63" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 44051 } }
I20260926 06:19:32.972998 24741 leader_election.cc:288] T 00000000000000000000000000000000 P 724e9816f07f4c0abb83624e37850698 [CANDIDATE]: Term 1 election: Requested vote from peers 148e04781dea445999e20ed86b870db8 (127.2.110.253:45513), 3cc49582c3a34406b0ca4ab50ac6fc63 (127.2.110.252:44051)
I20260926 06:19:32.973488 24891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "724e9816f07f4c0abb83624e37850698" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3cc49582c3a34406b0ca4ab50ac6fc63"
I20260926 06:19:32.973642 24891 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3cc49582c3a34406b0ca4ab50ac6fc63 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:19:32.974825 24891 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3cc49582c3a34406b0ca4ab50ac6fc63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 724e9816f07f4c0abb83624e37850698 in term 1.
I20260926 06:19:32.975096 24717 leader_election.cc:302] T 00000000000000000000000000000000 P 724e9816f07f4c0abb83624e37850698 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cc49582c3a34406b0ca4ab50ac6fc63, 724e9816f07f4c0abb83624e37850698; no voters:
W20260926 06:19:32.976675 24957 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 724e9816f07f4c0abb83624e37850698: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 06:19:32.976856 24957 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 06:19:32.977227 24951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 724e9816f07f4c0abb83624e37850698 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "724e9816f07f4c0abb83624e37850698" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 33011 } } peers { permanent_uuid: "148e04781dea445999e20ed86b870db8" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 45513 } } peers { permanent_uuid: "3cc49582c3a34406b0ca4ab50ac6fc63" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 44051 } } }
I20260926 06:19:32.977352 24951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 724e9816f07f4c0abb83624e37850698 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 06:19:32.977407 24951 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 724e9816f07f4c0abb83624e37850698 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:19:32.977450 24951 raft_consensus.cc:686] T 00000000000000000000000000000000 P 724e9816f07f4c0abb83624e37850698 [term 1 LEADER]: Becoming Leader. State: Replica: 724e9816f07f4c0abb83624e37850698, State: Running, Role: LEADER
I20260926 06:19:32.977525 24951 consensus_queue.cc:237] T 00000000000000000000000000000000 P 724e9816f07f4c0abb83624e37850698 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "724e9816f07f4c0abb83624e37850698" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 33011 } } peers { permanent_uuid: "148e04781dea445999e20ed86b870db8" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 45513 } } peers { permanent_uuid: "3cc49582c3a34406b0ca4ab50ac6fc63" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 44051 } }
I20260926 06:19:32.978018 24951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 724e9816f07f4c0abb83624e37850698 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 724e9816f07f4c0abb83624e37850698. Latest consensus state: current_term: 1 leader_uuid: "724e9816f07f4c0abb83624e37850698" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "724e9816f07f4c0abb83624e37850698" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 33011 } } peers { permanent_uuid: "148e04781dea445999e20ed86b870db8" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 45513 } } peers { permanent_uuid: "3cc49582c3a34406b0ca4ab50ac6fc63" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 44051 } } }
I20260926 06:19:32.978106 24951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 724e9816f07f4c0abb83624e37850698 [sys.catalog]: This master's current role is: LEADER
I20260926 06:19:32.978308 24960 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:19:32.978490 24960 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:19:32.977165 24699 master_runner.cc:186] Error getting master registration for 127.2.110.253:45513: OK, instance_id { permanent_uuid: "148e04781dea445999e20ed86b870db8" instance_seqno: 1790403572769304 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 06:19:32.980556 24699 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 06:19:32.983366 24770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:32.974336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59204 (local address 127.2.110.254:33011)
0926 06:19:32.974382 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:32.974384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:32.975022 (+ 638us) server_negotiation.cc:407] Connection header received
0926 06:19:32.975177 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:32.975178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:32.975202 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:32.975236 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:32.976158 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:32.976865 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:32.977956 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:32.978311 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:32.978352 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:32.980051 (+ 1699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:32.980052 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:32.980053 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:19:32.980060 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:32.981962 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:32.982452 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:32.982453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:32.982453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:32.982512 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:32.982728 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:32.982728 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:32.982729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:32.982867 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:32.982967 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:32.983181 (+ 214us) server_negotiation.cc:299] Negotiation successful
0926 06:19:32.983264 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 06:19:32.991669 24847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:32.974571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35012 (local address 127.2.110.253:45513)
0926 06:19:32.974641 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:32.974643 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:32.981508 (+ 6865us) server_negotiation.cc:407] Connection header received
0926 06:19:32.981622 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:32.981623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:32.981635 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:32.981670 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:32.982321 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:32.983080 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:32.986356 (+ 3276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:32.986668 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:32.986713 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:32.988368 (+ 1655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:32.988369 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:32.988369 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:19:32.988377 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:32.990286 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:32.990740 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:32.990741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:32.990741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:32.990800 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:32.991011 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:32.991012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:32.991012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:32.991166 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:32.991243 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:32.991468 (+ 225us) server_negotiation.cc:299] Negotiation successful
0926 06:19:32.991561 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 06:19:32.991983 24922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:32.974728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57740 (local address 127.2.110.252:44051)
0926 06:19:32.974765 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:32.974767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:32.983820 (+ 9053us) server_negotiation.cc:407] Connection header received
0926 06:19:32.983924 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:32.983925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:32.983938 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:32.983968 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:32.984636 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:32.985113 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:32.986253 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:32.986474 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:32.986562 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:32.988063 (+ 1501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:32.988065 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:32.988065 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:19:32.988073 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:32.990161 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:32.990868 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:32.990869 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:32.990869 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:32.990934 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:32.991231 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:32.991231 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:32.991232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:32.991370 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:32.991461 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:32.991800 (+ 339us) server_negotiation.cc:299] Negotiation successful
0926 06:19:32.991905 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 06:19:32.994531 24891 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3cc49582c3a34406b0ca4ab50ac6fc63 [term 1 FOLLOWER]: Refusing update from remote peer 724e9816f07f4c0abb83624e37850698: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:19:32.994926 24938 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 724e9816f07f4c0abb83624e37850698 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cc49582c3a34406b0ca4ab50ac6fc63" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 44051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:19:32.996331 24924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cc49582c3a34406b0ca4ab50ac6fc63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 724e9816f07f4c0abb83624e37850698. Latest consensus state: current_term: 1 leader_uuid: "724e9816f07f4c0abb83624e37850698" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "724e9816f07f4c0abb83624e37850698" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 33011 } } peers { permanent_uuid: "148e04781dea445999e20ed86b870db8" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 45513 } } peers { permanent_uuid: "3cc49582c3a34406b0ca4ab50ac6fc63" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 44051 } } }
I20260926 06:19:32.996430 24924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cc49582c3a34406b0ca4ab50ac6fc63 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 06:19:33.005353 24842 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.110.252" port: 44051 } attempt: 3
Sep 26 06:19:33 dist-test-slave-hndd krb5kdc[24686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403572, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.253@KRBTEST.COM for kudu/127.2.110.252@KRBTEST.COM
I20260926 06:19:33.011602 24938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 724e9816f07f4c0abb83624e37850698 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "724e9816f07f4c0abb83624e37850698" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "724e9816f07f4c0abb83624e37850698" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 33011 } } peers { permanent_uuid: "148e04781dea445999e20ed86b870db8" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 45513 } } peers { permanent_uuid: "3cc49582c3a34406b0ca4ab50ac6fc63" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 44051 } } }
I20260926 06:19:33.011734 24938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 724e9816f07f4c0abb83624e37850698 [sys.catalog]: This master's current role is: LEADER
I20260926 06:19:33.012310 24924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cc49582c3a34406b0ca4ab50ac6fc63 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "724e9816f07f4c0abb83624e37850698" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "724e9816f07f4c0abb83624e37850698" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 33011 } } peers { permanent_uuid: "148e04781dea445999e20ed86b870db8" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 45513 } } peers { permanent_uuid: "3cc49582c3a34406b0ca4ab50ac6fc63" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 44051 } } }
I20260926 06:19:33.012405 24924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cc49582c3a34406b0ca4ab50ac6fc63 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 06:19:33.012588 24961 mvcc.cc:205] Tried to move back new op lower bound from 7333493034982215680 to 7333493034916962304. Current Snapshot: MvccSnapshot[applied={T|T < 7333493034982215680}]
I20260926 06:19:33.014248 24922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:33.005889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57744 (local address 127.2.110.252:44051)
0926 06:19:33.005946 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:33.005948 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:33.005956 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:19:33.005997 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:33.005998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:33.006010 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:33.006060 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:33.006719 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:33.007502 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:33.008472 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:33.008773 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:33.008836 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:33.011147 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:33.011149 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:33.011149 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:19:33.011168 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:33.012974 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:33.013407 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:33.013408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:33.013408 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:33.013466 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:33.013669 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:33.013670 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:33.013671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:33.013828 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:33.013891 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:33.014053 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 06:19:33.014142 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 06:19:33.014242 24844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:33.005671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.252:44051 (local address 127.0.0.1:57744)
0926 06:19:33.005757 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:33.005770 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:33.005811 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:33.006077 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:19:33.006079 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:19:33.006293 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:19:33.006350 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:33.006352 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:33.007571 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:33.007572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:33.008396 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:33.008398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:33.008835 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:33.008836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:33.008896 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:33.009436 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:19:33.009443 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:19:33.011046 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:19:33.013068 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:19:33.013069 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:19:33.013070 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:19:33.013250 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:19:33.013532 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:19:33.013532 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:19:33.013533 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:19:33.013611 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:19:33.013887 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:19:33.013888 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:19:33.013935 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 06:19:33.014028 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 06:19:33.014138 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 06:19:33.016353 24842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 148e04781dea445999e20ed86b870db8: Bootstrap starting.
I20260926 06:19:33.017117 24842 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 148e04781dea445999e20ed86b870db8: Neither blocks nor log segments found. Creating new log.
I20260926 06:19:33.017426 24962 mvcc.cc:205] Tried to move back new op lower bound from 7333493034982215680 to 7333493034916962304. Current Snapshot: MvccSnapshot[applied={T|T < 7333493034982215680}]
I20260926 06:19:33.017469 24842 log.cc:824] T 00000000000000000000000000000000 P 148e04781dea445999e20ed86b870db8: Log is configured to *not* fsync() on all Append() calls
I20260926 06:19:33.018340 24842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 148e04781dea445999e20ed86b870db8: No bootstrap required, opened a new log
I20260926 06:19:33.018399 24960 catalog_manager.cc:1347] Generated new cluster ID: 8275fd8bf265493888391717863db8f3
I20260926 06:19:33.018446 24960 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:19:33.020269 24842 raft_consensus.cc:348] T 00000000000000000000000000000000 P 148e04781dea445999e20ed86b870db8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "724e9816f07f4c0abb83624e37850698" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 33011 } } peers { permanent_uuid: "148e04781dea445999e20ed86b870db8" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 45513 } } peers { permanent_uuid: "3cc49582c3a34406b0ca4ab50ac6fc63" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 44051 } }
I20260926 06:19:33.020409 24842 raft_consensus.cc:729] T 00000000000000000000000000000000 P 148e04781dea445999e20ed86b870db8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148e04781dea445999e20ed86b870db8, State: Initialized, Role: FOLLOWER
I20260926 06:19:33.020478 24842 consensus_queue.cc:260] T 00000000000000000000000000000000 P 148e04781dea445999e20ed86b870db8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "724e9816f07f4c0abb83624e37850698" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 33011 } } peers { permanent_uuid: "148e04781dea445999e20ed86b870db8" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 45513 } } peers { permanent_uuid: "3cc49582c3a34406b0ca4ab50ac6fc63" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 44051 } }
I20260926 06:19:33.020807 24842 sys_catalog.cc:564] T 00000000000000000000000000000000 P 148e04781dea445999e20ed86b870db8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:19:33.022430 24815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 148e04781dea445999e20ed86b870db8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:19:33.022500 24816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "724e9816f07f4c0abb83624e37850698" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "148e04781dea445999e20ed86b870db8"
I20260926 06:19:33.023401 24815 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 148e04781dea445999e20ed86b870db8 [term 1 FOLLOWER]: Refusing update from remote peer 724e9816f07f4c0abb83624e37850698: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:19:33.023900 24951 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 724e9816f07f4c0abb83624e37850698 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148e04781dea445999e20ed86b870db8" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 45513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:19:33.030769 24816 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 148e04781dea445999e20ed86b870db8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 724e9816f07f4c0abb83624e37850698 in term 1.
W20260926 06:19:33.033902 24978 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 148e04781dea445999e20ed86b870db8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 06:19:33.033978 24978 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 06:19:33.034103 24967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 148e04781dea445999e20ed86b870db8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "724e9816f07f4c0abb83624e37850698" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "724e9816f07f4c0abb83624e37850698" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 33011 } } peers { permanent_uuid: "148e04781dea445999e20ed86b870db8" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 45513 } } peers { permanent_uuid: "3cc49582c3a34406b0ca4ab50ac6fc63" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 44051 } } }
I20260926 06:19:33.034173 24967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 148e04781dea445999e20ed86b870db8 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 06:19:33.034228 24967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 148e04781dea445999e20ed86b870db8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 724e9816f07f4c0abb83624e37850698. Latest consensus state: current_term: 1 leader_uuid: "724e9816f07f4c0abb83624e37850698" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "724e9816f07f4c0abb83624e37850698" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 33011 } } peers { permanent_uuid: "148e04781dea445999e20ed86b870db8" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 45513 } } peers { permanent_uuid: "3cc49582c3a34406b0ca4ab50ac6fc63" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 44051 } } }
I20260926 06:19:33.034277 24967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 148e04781dea445999e20ed86b870db8 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 06:19:33.034310 24967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 148e04781dea445999e20ed86b870db8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "724e9816f07f4c0abb83624e37850698" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "724e9816f07f4c0abb83624e37850698" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 33011 } } peers { permanent_uuid: "148e04781dea445999e20ed86b870db8" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 45513 } } peers { permanent_uuid: "3cc49582c3a34406b0ca4ab50ac6fc63" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 44051 } } }
I20260926 06:19:33.034359 24967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 148e04781dea445999e20ed86b870db8 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 06:19:33.039906 24980 mvcc.cc:205] Tried to move back new op lower bound from 7333493034982215680 to 7333493034916962304. Current Snapshot: MvccSnapshot[applied={T|T < 7333493034982215680}]
I20260926 06:19:33.048718 24951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 724e9816f07f4c0abb83624e37850698 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "724e9816f07f4c0abb83624e37850698" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "724e9816f07f4c0abb83624e37850698" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 33011 } } peers { permanent_uuid: "148e04781dea445999e20ed86b870db8" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 45513 } } peers { permanent_uuid: "3cc49582c3a34406b0ca4ab50ac6fc63" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 44051 } } }
I20260926 06:19:33.048851 24951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 724e9816f07f4c0abb83624e37850698 [sys.catalog]: This master's current role is: LEADER
I20260926 06:19:33.049983 24960 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:19:33.050729 24960 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:19:33.059055 24960 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 724e9816f07f4c0abb83624e37850698: Generated new TSK 0
I20260926 06:19:33.059414 24960 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 06:19:33.086771 24847 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:19:33.085008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35016 (local address 127.2.110.253:45513)
0926 06:19:33.085054 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:33.085056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:33.086046 (+ 990us) server_negotiation.cc:407] Connection header received
0926 06:19:33.086150 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:33.086151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:33.086162 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:33.086193 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:33.086697 (+ 504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35016: BlockingRecv error: recv got EOF from 127.0.0.1:35016 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 06:19:33.087616 24922 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:19:33.085427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57746 (local address 127.2.110.252:44051)
0926 06:19:33.085471 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:33.085473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:33.087056 (+ 1583us) server_negotiation.cc:407] Connection header received
0926 06:19:33.087201 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:33.087202 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:33.087215 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:33.087245 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:33.087550 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57746: BlockingRecv error: recv got EOF from 127.0.0.1:57746 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 06:19:33.088902 24770 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:19:33.085774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59210 (local address 127.2.110.254:33011)
0926 06:19:33.085813 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:33.085815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:33.087848 (+ 2033us) server_negotiation.cc:407] Connection header received
0926 06:19:33.087946 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:33.087947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:33.087958 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:33.087989 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:33.088806 (+ 817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59210: BlockingRecv error: recv got EOF from 127.0.0.1:59210 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 06:19:33.091881 24847 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:19:33.091038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35030 (local address 127.2.110.253:45513)
0926 06:19:33.091079 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:33.091081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:33.091319 (+ 238us) server_negotiation.cc:407] Connection header received
0926 06:19:33.091426 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:33.091427 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:33.091447 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:33.091477 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:33.091827 (+ 350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35030: BlockingRecv error: recv got EOF from 127.0.0.1:35030 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 06:19:33.095173 24847 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:19:33.094266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35032 (local address 127.2.110.253:45513)
0926 06:19:33.094303 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:33.094305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:33.094590 (+ 285us) server_negotiation.cc:407] Connection header received
0926 06:19:33.094709 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:33.094710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:33.094722 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:33.094753 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:33.095119 (+ 366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35032: BlockingRecv error: recv got EOF from 127.0.0.1:35032 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260926 06:19:33.099151 24847 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:19:33.098301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35044 (local address 127.2.110.253:45513)
0926 06:19:33.098339 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:33.098341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:33.098592 (+ 251us) server_negotiation.cc:407] Connection header received
0926 06:19:33.098709 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:33.098710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:33.098722 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:33.098752 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:33.099095 (+ 343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35044: BlockingRecv error: recv got EOF from 127.0.0.1:35044 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 06:19:33.105986 24847 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:19:33.104425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35054 (local address 127.2.110.253:45513)
0926 06:19:33.104482 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:33.104484 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:33.104789 (+ 305us) server_negotiation.cc:407] Connection header received
0926 06:19:33.104899 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:33.104900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:33.104913 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:33.104945 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:33.105890 (+ 945us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35054: BlockingRecv error: recv got EOF from 127.0.0.1:35054 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260926 06:19:33.116432 24847 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:19:33.115397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35062 (local address 127.2.110.253:45513)
0926 06:19:33.115446 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:33.115448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:33.115767 (+ 319us) server_negotiation.cc:407] Connection header received
0926 06:19:33.115882 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:33.115883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:33.115918 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:33.115952 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:33.116367 (+ 415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35062: BlockingRecv error: recv got EOF from 127.0.0.1:35062 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 06:19:33.134886 24847 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:19:33.133868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35078 (local address 127.2.110.253:45513)
0926 06:19:33.133914 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:33.133917 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:33.134214 (+ 297us) server_negotiation.cc:407] Connection header received
0926 06:19:33.134330 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:33.134331 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:33.134345 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:33.134375 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:33.134818 (+ 443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35078: BlockingRecv error: recv got EOF from 127.0.0.1:35078 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 06:19:33.169394 24847 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:19:33.168405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35094 (local address 127.2.110.253:45513)
0926 06:19:33.168458 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:33.168460 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:33.168768 (+ 308us) server_negotiation.cc:407] Connection header received
0926 06:19:33.168870 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:33.168871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:33.168884 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:33.168914 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:33.169301 (+ 387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35094: BlockingRecv error: recv got EOF from 127.0.0.1:35094 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260926 06:19:33.235494 24847 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:19:33.234858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35098 (local address 127.2.110.253:45513)
0926 06:19:33.234951 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:33.234953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:33.234979 (+ 26us) server_negotiation.cc:407] Connection header received
0926 06:19:33.235026 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:33.235027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:33.235038 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:33.235107 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:33.235421 (+ 314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35098: BlockingRecv error: recv got EOF from 127.0.0.1:35098 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260926 06:19:33.365299 24847 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:19:33.364653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35106 (local address 127.2.110.253:45513)
0926 06:19:33.364751 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:33.364753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:33.364770 (+ 17us) server_negotiation.cc:407] Connection header received
0926 06:19:33.364845 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:33.364845 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:33.364856 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:33.364909 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:33.365240 (+ 331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35106: BlockingRecv error: recv got EOF from 127.0.0.1:35106 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260926 06:19:33.623370 24847 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:19:33.622560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35118 (local address 127.2.110.253:45513)
0926 06:19:33.622655 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:33.622657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:33.622778 (+ 121us) server_negotiation.cc:407] Connection header received
0926 06:19:33.622871 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:33.622872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:33.622884 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:33.622938 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:33.623296 (+ 358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35118: BlockingRecv error: recv got EOF from 127.0.0.1:35118 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 06:19:33.963835 24935 catalog_manager.cc:1259] Loaded cluster ID: 8275fd8bf265493888391717863db8f3
I20260926 06:19:33.963922 24935 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3cc49582c3a34406b0ca4ab50ac6fc63: loading cluster ID for follower catalog manager: success
I20260926 06:19:33.964722 24935 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3cc49582c3a34406b0ca4ab50ac6fc63: acquiring CA information for follower catalog manager: success
I20260926 06:19:33.964972 24935 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3cc49582c3a34406b0ca4ab50ac6fc63: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 06:19:34.034463 24978 catalog_manager.cc:1259] Loaded cluster ID: 8275fd8bf265493888391717863db8f3
I20260926 06:19:34.034616 24978 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 148e04781dea445999e20ed86b870db8: loading cluster ID for follower catalog manager: success
I20260926 06:19:34.035455 24978 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 148e04781dea445999e20ed86b870db8: acquiring CA information for follower catalog manager: success
I20260926 06:19:34.035778 24978 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 148e04781dea445999e20ed86b870db8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 06:19:34.139183 25051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:34.136605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35130 (local address 127.2.110.253:45513)
0926 06:19:34.136712 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:34.136714 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:34.136795 (+ 81us) server_negotiation.cc:407] Connection header received
0926 06:19:34.136847 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:34.136848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:34.136898 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:34.136972 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:19:34.137138 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:34.137656 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:34.138405 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:34.138610 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:34.138667 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:34.138804 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:19:34.138920 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:19:34.138981 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:34.139028 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 06:19:34.139047 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":54,"threads_started":1}
I20260926 06:19:34.143853 25057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:34.141392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57750 (local address 127.2.110.252:44051)
0926 06:19:34.141528 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:34.141530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:34.141540 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:19:34.141597 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:34.141597 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:34.141633 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:34.141708 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:19:34.141845 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:34.142317 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:34.143135 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:34.143333 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:34.143383 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:34.143509 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:19:34.143608 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:19:34.143661 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:34.143707 (+ 46us) server_negotiation.cc:299] Negotiation successful
0926 06:19:34.143723 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":91,"threads_started":1}
I20260926 06:19:34.148789 25057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:34.146067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57760 (local address 127.2.110.252:44051)
0926 06:19:34.146125 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:34.146127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:34.146206 (+ 79us) server_negotiation.cc:407] Connection header received
0926 06:19:34.146274 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:34.146275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:34.146287 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:34.146338 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:19:34.146463 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:34.146896 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:34.147958 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:34.148246 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:34.148310 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:34.148453 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:19:34.148486 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:19:34.148535 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:34.148664 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 06:19:34.148694 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 06:19:34.148907 25051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:34.145860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35140 (local address 127.2.110.253:45513)
0926 06:19:34.145920 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:34.145922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:34.145932 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:19:34.145964 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:34.145965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:34.145979 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:34.146054 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:19:34.146198 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:34.147103 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:34.148155 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:34.148431 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:34.148497 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:34.148628 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:19:34.148662 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:19:34.148706 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:34.148809 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 06:19:34.148834 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 06:19:34.149216 25064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:34.145734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59222 (local address 127.2.110.254:33011)
0926 06:19:34.145930 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:34.145932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:34.145940 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:19:34.145980 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:34.145980 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:34.146022 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:34.146196 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:19:34.146501 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:34.147331 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:34.148224 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:34.148531 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:34.148590 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:34.148812 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:19:34.148929 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:19:34.148993 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:34.149108 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 06:19:34.149136 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":54,"threads_started":1}
I20260926 06:19:34.150799 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 24699
I20260926 06:19:34.156399 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 24774
I20260926 06:19:34.161598 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 24849
2026-09-26T06:19:34Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2856 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:19:34 dist-test-slave-hndd krb5kdc[25068](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:19:34 dist-test-slave-hndd krb5kdc[25068](info): set up 2 sockets
Sep 26 06:19:34 dist-test-slave-hndd krb5kdc[25068](info): commencing operation
krb5kdc: starting...
W20260926 06:19:35.229707 2491 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/krb5kdc/test-user.keytab.
Sep 26 06:19:35 dist-test-slave-hndd krb5kdc[25068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:19:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:19:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
Entry for principal kudu/127.2.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab.
I20260926 06:19:35.359393 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:43265
--webserver_interface=127.2.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:40797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.110.254:43265,127.2.110.253:34687,127.2.110.252:42729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:19:35.440186 25081 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 06:19:35.440414 25081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:19:35.440433 25081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:19:35.440454 25081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:19:35.441553 25081 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:19:35.441592 25081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:19:35.441606 25081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:19:35.441619 25081 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:19:35.441632 25081 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:19:35.443091 25081 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:40797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.110.254:43265,127.2.110.253:34687,127.2.110.252:42729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.254.keytab
--principal=kudu/127.2.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.254:43265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:19:35.443287 25081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:19:35.443459 25081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:19:35.448585 25087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:35.448591 25089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:35.448585 25086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:35.448594 25081 server_base.cc:1034] running on GCE node
I20260926 06:19:35.449038 25081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:19:35.449383 25081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:19:35.450559 25081 hybrid_clock.cc:648] HybridClock initialized: now 1790403575450540 us; error 35 us; skew 500 ppm
Sep 26 06:19:35 dist-test-slave-hndd krb5kdc[25068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403575, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:19:35.458328 25081 init.cc:377] Logged in from keytab as kudu/127.2.110.254@KRBTEST.COM (short username kudu)
I20260926 06:19:35.458904 25081 webserver.cc:460] Webserver started at http://127.2.110.254:45411/ using document root <none> and password file <none>
I20260926 06:19:35.459169 25081 fs_manager.cc:359] Metadata directory not provided
I20260926 06:19:35.459244 25081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:19:35.459398 25081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:19:35.460587 25081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/minicluster-data/master-0/data/instance:
uuid: "cdd3f8d1dd0b4bf8aa62d31067735c4b"
format_stamp: "Formatted at 2026-09-26 06:19:35 on dist-test-slave-hndd"
server_key: "b4788a7716afb15e2098d3da56141707"
server_key_iv: "554f1c6508388ecd311a841eb239eba8"
server_key_version: "encryptionkey@0"
I20260926 06:19:35.460979 25081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/minicluster-data/master-0/wal/instance:
uuid: "cdd3f8d1dd0b4bf8aa62d31067735c4b"
format_stamp: "Formatted at 2026-09-26 06:19:35 on dist-test-slave-hndd"
server_key: "b4788a7716afb15e2098d3da56141707"
server_key_iv: "554f1c6508388ecd311a841eb239eba8"
server_key_version: "encryptionkey@0"
I20260926 06:19:35.462512 25081 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 06:19:35.463370 25096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:35.463552 25081 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:19:35.463657 25081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/minicluster-data/master-0/wal
uuid: "cdd3f8d1dd0b4bf8aa62d31067735c4b"
format_stamp: "Formatted at 2026-09-26 06:19:35 on dist-test-slave-hndd"
server_key: "b4788a7716afb15e2098d3da56141707"
server_key_iv: "554f1c6508388ecd311a841eb239eba8"
server_key_version: "encryptionkey@0"
I20260926 06:19:35.463730 25081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:19:35.501040 25081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:19:35.503347 25081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:19:35.503469 25081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:19:35.507396 25081 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.254:43265
I20260926 06:19:35.507462 25148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.254:43265 every 8 connection(s)
I20260926 06:19:35.507827 25081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/minicluster-data/master-0/data/info.pb
I20260926 06:19:35.509091 25149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:19:35.510722 25149 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.110.254" port: 43265 } has no permanent_uuid. Determining permanent_uuid...
I20260926 06:19:35.515424 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 25081
I20260926 06:19:35.515519 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/minicluster-data/master-0/wal/instance
I20260926 06:19:35.515755 2491 external_mini_cluster.cc:1428] Setting key 9e52a05d3c859b740ab2f9f07c3e3d2d
Sep 26 06:19:35 dist-test-slave-hndd krb5kdc[25068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403575, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.254@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:19:35.522691 25152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:35.511673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43068 (local address 127.2.110.254:43265)
0926 06:19:35.511817 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:35.511821 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:35.511854 (+ 33us) server_negotiation.cc:407] Connection header received
0926 06:19:35.512390 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:35.512392 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:35.512412 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:35.512516 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:35.513079 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:35.513961 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:35.514835 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:35.515145 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:35.515265 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:35.518425 (+ 3160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:35.518436 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:35.518436 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:19:35.518445 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:35.520549 (+ 2104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:35.521162 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:35.521168 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:35.521169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:35.521322 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:35.521779 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:35.521781 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:35.521781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:35.522060 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:35.522144 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:35.522397 (+ 253us) server_negotiation.cc:299] Negotiation successful
0926 06:19:35.522509 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":65,"threads_started":1}
I20260926 06:19:35.522799 25151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:35.511490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:43265 (local address 127.0.0.1:43068)
0926 06:19:35.511652 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:35.511688 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:35.512227 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:35.512520 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:19:35.512523 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:19:35.512824 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:19:35.512975 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:35.512980 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:35.514072 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:35.514073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:35.514745 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:35.514747 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:35.515273 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:35.515275 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:35.515341 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:35.515981 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:19:35.515992 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:19:35.518256 (+ 2264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:19:35.520652 (+ 2396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:19:35.520659 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:19:35.520663 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:19:35.520964 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:19:35.521497 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:19:35.521499 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:19:35.521499 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:19:35.521650 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:19:35.522144 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:19:35.522152 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:19:35.522244 (+ 92us) client_negotiation.cc:769] Sending connection context
0926 06:19:35.522360 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 06:19:35.522627 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":69,"thread_start_us":84,"threads_started":1}
I20260926 06:19:35.523656 25149 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.110.253" port: 34687 } has no permanent_uuid. Determining permanent_uuid...
I20260926 06:19:35.524056 25151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:35.523912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.253:34687 (local address 127.0.0.1:33574)
0926 06:19:35.523969 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:35.524024 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.253:34687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 06:19:35.524186 25099 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.110.253:34687: connect: Connection refused (error 111)
I20260926 06:19:35.524631 25151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:35.524522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.253:34687 (local address 127.0.0.1:33586)
0926 06:19:35.524571 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:35.524601 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.253:34687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 06:19:35.524755 25149 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.110.253:34687: Network error: Client connection negotiation failed: client connection to 127.2.110.253:34687: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.2.110.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.253.keytab.
Entry for principal kudu/127.2.110.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.253.keytab.
I20260926 06:19:35.545420 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.253:34687
--webserver_interface=127.2.110.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.253.keytab
--principal=kudu/127.2.110.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:40797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.110.254:43265,127.2.110.253:34687,127.2.110.252:42729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 06:19:35.574023 25149 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.110.253" port: 34687 } attempt: 1
I20260926 06:19:35.574589 25151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:35.574430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.253:34687 (local address 127.0.0.1:33598)
0926 06:19:35.574500 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:35.574555 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.253:34687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 06:19:35.574959 25151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:35.574872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.253:34687 (local address 127.0.0.1:33608)
0926 06:19:35.574912 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:35.574941 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.253:34687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260926 06:19:35.575152 25149 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.110.253:34687: Network error: Client connection negotiation failed: client connection to 127.2.110.253:34687: connect: Connection refused (error 111)
W20260926 06:19:35.632176 25156 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 06:19:35.632423 25156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:19:35.632443 25156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:19:35.632467 25156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:19:35.633723 25156 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:19:35.633773 25156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:19:35.633787 25156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:19:35.633801 25156 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:19:35.633813 25156 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:19:35.635439 25156 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:40797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.110.254:43265,127.2.110.253:34687,127.2.110.252:42729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.253.keytab
--principal=kudu/127.2.110.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.253:34687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.110.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:19:35.635715 25156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:19:35.635953 25156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:19:35.640738 25164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:35.640766 25162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:35.640895 25156 server_base.cc:1034] running on GCE node
W20260926 06:19:35.640798 25161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:35.641213 25156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:19:35.641539 25156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:19:35.642719 25156 hybrid_clock.cc:648] HybridClock initialized: now 1790403575642708 us; error 34 us; skew 500 ppm
I20260926 06:19:35.643301 25149 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.110.253" port: 34687 } attempt: 2
I20260926 06:19:35.643707 25151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:35.643554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.253:34687 (local address 127.0.0.1:33614)
0926 06:19:35.643624 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:35.643665 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.253:34687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 06:19:35.644116 25151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:35.644028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.253:34687 (local address 127.0.0.1:33624)
0926 06:19:35.644070 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:35.644095 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.253:34687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 06:19:35.644222 25149 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.110.253:34687: Network error: Client connection negotiation failed: client connection to 127.2.110.253:34687: connect: Connection refused (error 111)
Sep 26 06:19:35 dist-test-slave-hndd krb5kdc[25068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403575, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:19:35.645625 25156 init.cc:377] Logged in from keytab as kudu/127.2.110.253@KRBTEST.COM (short username kudu)
I20260926 06:19:35.646095 25156 webserver.cc:460] Webserver started at http://127.2.110.253:42615/ using document root <none> and password file <none>
I20260926 06:19:35.646332 25156 fs_manager.cc:359] Metadata directory not provided
I20260926 06:19:35.646399 25156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:19:35.646544 25156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:19:35.647599 25156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/minicluster-data/master-1/data/instance:
uuid: "778f32bf95a648d5bff3275adb49ceb9"
format_stamp: "Formatted at 2026-09-26 06:19:35 on dist-test-slave-hndd"
server_key: "9497645be771453a935e268c854e453b"
server_key_iv: "fc26d8eb75c1effa38fc3ba99d755c68"
server_key_version: "encryptionkey@0"
I20260926 06:19:35.647960 25156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/minicluster-data/master-1/wal/instance:
uuid: "778f32bf95a648d5bff3275adb49ceb9"
format_stamp: "Formatted at 2026-09-26 06:19:35 on dist-test-slave-hndd"
server_key: "9497645be771453a935e268c854e453b"
server_key_iv: "fc26d8eb75c1effa38fc3ba99d755c68"
server_key_version: "encryptionkey@0"
I20260926 06:19:35.649297 25156 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 06:19:35.650128 25171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:35.650324 25156 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:19:35.650434 25156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/minicluster-data/master-1/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/minicluster-data/master-1/wal
uuid: "778f32bf95a648d5bff3275adb49ceb9"
format_stamp: "Formatted at 2026-09-26 06:19:35 on dist-test-slave-hndd"
server_key: "9497645be771453a935e268c854e453b"
server_key_iv: "fc26d8eb75c1effa38fc3ba99d755c68"
server_key_version: "encryptionkey@0"
I20260926 06:19:35.650521 25156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:19:35.663309 25156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:19:35.665642 25156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:19:35.665817 25156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:19:35.670914 25156 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.253:34687
I20260926 06:19:35.670970 25223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.253:34687 every 8 connection(s)
I20260926 06:19:35.671314 25156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/minicluster-data/master-1/data/info.pb
I20260926 06:19:35.672271 25224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:19:35.673906 25224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.110.254" port: 43265 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 06:19:35 dist-test-slave-hndd krb5kdc[25068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403575, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.253@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:19:35.681030 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 25156
I20260926 06:19:35.681154 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/minicluster-data/master-1/wal/instance
I20260926 06:19:35.681375 2491 external_mini_cluster.cc:1428] Setting key bebd4e71cd5b6f10b9740ca6af646f11
I20260926 06:19:35.683003 25152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:35.674654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43070 (local address 127.2.110.254:43265)
0926 06:19:35.674735 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:35.674737 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:35.674993 (+ 256us) server_negotiation.cc:407] Connection header received
0926 06:19:35.675440 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:35.675441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:35.675455 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:35.675513 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:35.676111 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:35.676681 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:35.677700 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:35.677920 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:35.677983 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:35.680079 (+ 2096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:35.680080 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:35.680080 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:19:35.680087 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:35.681625 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:35.681995 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:35.681996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:35.681997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:35.682071 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:35.682291 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:35.682292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:35.682292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:35.682447 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:35.682516 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:35.682811 (+ 295us) server_negotiation.cc:299] Negotiation successful
0926 06:19:35.682910 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 06:19:35.683192 25226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:35.674639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:43265 (local address 127.0.0.1:43070)
0926 06:19:35.674852 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:35.674889 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:35.675342 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:35.675577 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:19:35.675579 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:19:35.675828 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:19:35.676025 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:35.676028 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:35.676759 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:35.676760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:35.677620 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:35.677623 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:35.678008 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:35.678009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:35.678103 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:35.678650 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:19:35.678658 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:19:35.679986 (+ 1328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:19:35.681702 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:19:35.681708 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:19:35.681709 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:19:35.681921 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:19:35.682136 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:19:35.682136 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:19:35.682137 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:19:35.682231 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:19:35.682511 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:19:35.682515 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:19:35.682628 (+ 113us) client_negotiation.cc:769] Sending connection context
0926 06:19:35.682779 (+ 151us) client_negotiation.cc:240] Negotiation successful
0926 06:19:35.682951 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":101,"threads_started":1}
I20260926 06:19:35.683784 25224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.110.253" port: 34687 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 06:19:35 dist-test-slave-hndd krb5kdc[25068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403575, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.253@KRBTEST.COM for kudu/127.2.110.253@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.110.252@KRBTEST.COM; defaulting to no policy
I20260926 06:19:35.691654 25226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:35.684042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.253:34687 (local address 127.0.0.1:33634)
0926 06:19:35.684248 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:35.684259 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:35.684313 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:35.684475 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:19:35.684476 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:19:35.684654 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:19:35.684726 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:35.684728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:35.685586 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:35.685587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:35.686461 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:35.686464 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:35.686948 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:35.686949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:35.687035 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:35.687415 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:19:35.687420 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:19:35.688709 (+ 1289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:19:35.690544 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:19:35.690545 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:19:35.690546 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:19:35.690706 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:19:35.690913 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:19:35.690914 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:19:35.690914 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:19:35.690967 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:19:35.691324 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:19:35.691325 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:19:35.691365 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 06:19:35.691436 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 06:19:35.691538 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190}
I20260926 06:19:35.691784 25228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:35.684105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33634 (local address 127.2.110.253:34687)
0926 06:19:35.684256 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:35.684260 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:35.684306 (+ 46us) server_negotiation.cc:407] Connection header received
0926 06:19:35.684377 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:35.684380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:35.684393 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:35.684463 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:35.684798 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:35.685502 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:35.686567 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:35.686869 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:35.686933 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:35.688804 (+ 1871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:35.688813 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:35.688813 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:19:35.688819 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:35.690428 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:35.690793 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:35.690796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:35.690796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:35.690852 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:35.691048 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:35.691049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:35.691050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:35.691216 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:35.691345 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:35.691466 (+ 121us) server_negotiation.cc:299] Negotiation successful
0926 06:19:35.691607 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":126,"thread_start_us":89,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.110.252@KRBTEST.COM" created.
I20260926 06:19:35.692408 25224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.110.252" port: 42729 } has no permanent_uuid. Determining permanent_uuid...
I20260926 06:19:35.692787 25226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:35.692665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.252:42729 (local address 127.0.0.1:54938)
0926 06:19:35.692714 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:35.692763 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.252:42729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 06:19:35.692917 25174 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.110.252:42729: connect: Connection refused (error 111)
I20260926 06:19:35.693326 25226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:35.693236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.252:42729 (local address 127.0.0.1:54942)
0926 06:19:35.693277 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:35.693307 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.252:42729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260926 06:19:35.693449 25224 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.110.252:42729: Network error: Client connection negotiation failed: client connection to 127.2.110.252:42729: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.110.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.252.keytab.
Entry for principal kudu/127.2.110.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.252.keytab.
I20260926 06:19:35.708039 2491 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgcQD_/build/release/bin/kudu
/tmp/dist-test-taskRgcQD_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.252:42729
--webserver_interface=127.2.110.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.252.keytab
--principal=kudu/127.2.110.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.110.212:40797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.110.254:43265,127.2.110.253:34687,127.2.110.252:42729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 06:19:35.735376 25149 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.110.253" port: 34687 } attempt: 3
Sep 26 06:19:35 dist-test-slave-hndd krb5kdc[25068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403575, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.254@KRBTEST.COM for kudu/127.2.110.253@KRBTEST.COM
I20260926 06:19:35.742587 25228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:35.735899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33648 (local address 127.2.110.253:34687)
0926 06:19:35.735975 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:35.735977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:35.735988 (+ 11us) server_negotiation.cc:407] Connection header received
0926 06:19:35.736018 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:35.736019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:35.736032 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:35.736100 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:35.736427 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:35.736924 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:35.737661 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:35.737882 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:35.737953 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:35.739920 (+ 1967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:35.739922 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:35.739923 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:19:35.739930 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:35.741448 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:35.741847 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:35.741848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:35.741848 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:35.741909 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:35.742118 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:35.742119 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:35.742119 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:35.742233 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:35.742287 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:35.742445 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 06:19:35.742514 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 06:19:35.742655 25151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:35.735819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.253:34687 (local address 127.0.0.1:33648)
0926 06:19:35.735876 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:35.735894 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:35.735955 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:35.736107 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:19:35.736108 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:19:35.736276 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:19:35.736341 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:35.736343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:35.737015 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:35.737017 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:35.737554 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:35.737556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:35.737960 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:35.737961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:35.738036 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:35.738497 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:19:35.738502 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:19:35.739801 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:19:35.741524 (+ 1723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:19:35.741525 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:19:35.741525 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:19:35.741750 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:19:35.741980 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:19:35.741981 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:19:35.741981 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:19:35.742047 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:19:35.742293 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:19:35.742294 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:19:35.742346 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 06:19:35.742421 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 06:19:35.742524 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 06:19:35.742671 25224 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.110.252" port: 42729 } attempt: 1
I20260926 06:19:35.743054 25149 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.110.252" port: 42729 } has no permanent_uuid. Determining permanent_uuid...
I20260926 06:19:35.743147 25226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:35.742907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.252:42729 (local address 127.0.0.1:54958)
0926 06:19:35.743076 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:35.743114 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.252:42729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
I20260926 06:19:35.743376 25151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:35.743275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.252:42729 (local address 127.0.0.1:54974)
0926 06:19:35.743324 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:35.743353 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.252:42729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 06:19:35.743489 25226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:35.743365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.252:42729 (local address 127.0.0.1:54990)
0926 06:19:35.743418 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:35.743460 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.252:42729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 06:19:35.743577 25224 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.110.252:42729: Network error: Client connection negotiation failed: client connection to 127.2.110.252:42729: connect: Connection refused (error 111)
I20260926 06:19:35.743878 25232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:35.743609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.252:42729 (local address 127.0.0.1:54994)
0926 06:19:35.743791 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:35.743830 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.252:42729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":105,"threads_started":1}
W20260926 06:19:35.743968 25149 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.110.252:42729: Network error: Client connection negotiation failed: client connection to 127.2.110.252:42729: connect: Connection refused (error 111)
I20260926 06:19:35.775102 25149 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.110.252" port: 42729 } attempt: 1
I20260926 06:19:35.775643 25232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:35.775459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.252:42729 (local address 127.0.0.1:55010)
0926 06:19:35.775534 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:35.775595 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.252:42729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 06:19:35.776017 25232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:35.775918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.252:42729 (local address 127.0.0.1:55014)
0926 06:19:35.775966 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:35.775996 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.252:42729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 06:19:35.776139 25149 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.110.252:42729: Network error: Client connection negotiation failed: client connection to 127.2.110.252:42729: connect: Connection refused (error 111)
W20260926 06:19:35.791312 25231 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 06:19:35.791517 25231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:19:35.791533 25231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:19:35.791581 25231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:19:35.792719 25231 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:19:35.792760 25231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:19:35.792774 25231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:19:35.792821 25231 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:19:35.792840 25231 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:19:35.794395 25231 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.110.212:40797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.110.254:43265,127.2.110.253:34687,127.2.110.252:42729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/krb5kdc/kudu_127.2.110.252.keytab
--principal=kudu/127.2.110.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.110.252:42729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.110.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:19:35.794646 25231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:19:35.794878 25231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:19:35.799953 25240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:35.799948 25237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:19:35.800005 25238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:19:35.800140 25231 server_base.cc:1034] running on GCE node
I20260926 06:19:35.800419 25231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:19:35.800668 25231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:19:35.801810 25231 hybrid_clock.cc:648] HybridClock initialized: now 1790403575801786 us; error 34 us; skew 500 ppm
Sep 26 06:19:35 dist-test-slave-hndd krb5kdc[25068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403575, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:19:35.804760 25231 init.cc:377] Logged in from keytab as kudu/127.2.110.252@KRBTEST.COM (short username kudu)
I20260926 06:19:35.805187 25231 webserver.cc:460] Webserver started at http://127.2.110.252:38001/ using document root <none> and password file <none>
I20260926 06:19:35.805505 25231 fs_manager.cc:359] Metadata directory not provided
I20260926 06:19:35.805580 25231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:19:35.805747 25231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:19:35.806936 25231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/minicluster-data/master-2/data/instance:
uuid: "2c6e5b98856445f2b999747ff1f36425"
format_stamp: "Formatted at 2026-09-26 06:19:35 on dist-test-slave-hndd"
server_key: "aad12f227185d3253d266a45ccf611e8"
server_key_iv: "de07100b1f6f49a550733a3031e425e5"
server_key_version: "encryptionkey@0"
I20260926 06:19:35.807341 25231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/minicluster-data/master-2/wal/instance:
uuid: "2c6e5b98856445f2b999747ff1f36425"
format_stamp: "Formatted at 2026-09-26 06:19:35 on dist-test-slave-hndd"
server_key: "aad12f227185d3253d266a45ccf611e8"
server_key_iv: "de07100b1f6f49a550733a3031e425e5"
server_key_version: "encryptionkey@0"
I20260926 06:19:35.808805 25231 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:19:35.809708 25247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:19:35.809906 25231 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:19:35.810019 25231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/minicluster-data/master-2/data,/tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/minicluster-data/master-2/wal
uuid: "2c6e5b98856445f2b999747ff1f36425"
format_stamp: "Formatted at 2026-09-26 06:19:35 on dist-test-slave-hndd"
server_key: "aad12f227185d3253d266a45ccf611e8"
server_key_iv: "de07100b1f6f49a550733a3031e425e5"
server_key_version: "encryptionkey@0"
I20260926 06:19:35.810096 25231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:19:35.811728 25224 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.110.252" port: 42729 } attempt: 2
I20260926 06:19:35.812232 25226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:35.812075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.252:42729 (local address 127.0.0.1:55020)
0926 06:19:35.812144 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:35.812191 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.252:42729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 06:19:35.812770 25226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:35.812631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.252:42729 (local address 127.0.0.1:55034)
0926 06:19:35.812716 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:35.812747 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.110.252:42729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260926 06:19:35.812885 25224 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.110.252:42729: Network error: Client connection negotiation failed: client connection to 127.2.110.252:42729: connect: Connection refused (error 111)
I20260926 06:19:35.826428 25231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:19:35.828550 25231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:19:35.828734 25231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:19:35.832784 25231 rpc_server.cc:307] RPC server started. Bound to: 127.2.110.252:42729
I20260926 06:19:35.832820 25299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.110.252:42729 every 8 connection(s)
I20260926 06:19:35.833248 25231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/minicluster-data/master-2/data/info.pb
I20260926 06:19:35.834527 25300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:19:35.836248 25300 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.110.254" port: 43265 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 06:19:35 dist-test-slave-hndd krb5kdc[25068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403575, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.252@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:19:35.843417 2491 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgcQD_/build/release/bin/kudu as pid 25231
I20260926 06:19:35.843532 2491 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgcQD_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403292369990-2491-0/minicluster-data/master-2/wal/instance
I20260926 06:19:35.843775 2491 external_mini_cluster.cc:1428] Setting key 80fb05085baff90f170c406fe6dc3bc2
I20260926 06:19:35.847137 25152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:35.837036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43076 (local address 127.2.110.254:43265)
0926 06:19:35.837152 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:35.837154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:35.837363 (+ 209us) server_negotiation.cc:407] Connection header received
0926 06:19:35.837853 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:35.837854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:35.837864 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:35.837936 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:35.838570 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:35.839059 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:35.840151 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:35.840342 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:35.840405 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:35.843040 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:35.843043 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:35.843043 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:19:35.843054 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:35.845245 (+ 2191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:35.845926 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:35.845927 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:35.845928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:35.845988 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:35.846302 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:35.846302 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:35.846303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:35.846416 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:35.846465 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:35.846869 (+ 404us) server_negotiation.cc:299] Negotiation successful
0926 06:19:35.847018 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 06:19:35.847146 25302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:35.836951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.254:43265 (local address 127.0.0.1:43076)
0926 06:19:35.837219 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:35.837252 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:35.837762 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:35.838008 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:19:35.838011 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:19:35.838280 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:19:35.838474 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:35.838478 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:35.839166 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:35.839167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:35.840046 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:35.840049 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:35.840443 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:35.840444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:35.840557 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:35.841123 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:19:35.841131 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:19:35.842901 (+ 1770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:19:35.845432 (+ 2531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:19:35.845439 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:19:35.845442 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:19:35.845800 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:19:35.846073 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:19:35.846074 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:19:35.846074 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:19:35.846223 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:19:35.846463 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:19:35.846468 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:19:35.846632 (+ 164us) client_negotiation.cc:769] Sending connection context
0926 06:19:35.846745 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 06:19:35.846920 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":106,"threads_started":1}
I20260926 06:19:35.848042 25300 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.110.253" port: 34687 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 06:19:35 dist-test-slave-hndd krb5kdc[25068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.110.252@KRBTEST.COM
I20260926 06:19:35.851294 25149 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.110.252" port: 42729 } attempt: 2
Sep 26 06:19:35 dist-test-slave-hndd krb5kdc[25068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403575, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.252@KRBTEST.COM for kudu/127.2.110.253@KRBTEST.COM
I20260926 06:19:35.855865 25304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:35.844292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55040 (local address 127.2.110.252:42729)
0926 06:19:35.844504 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:35.844510 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:35.844567 (+ 57us) server_negotiation.cc:407] Connection header received
0926 06:19:35.844661 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:35.844663 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:35.844678 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:35.844823 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:35.845637 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:35.846889 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:35.848232 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:35.848913 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:35.848997 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:35.851824 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:35.851833 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:35.851834 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:19:35.851840 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:35.854403 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:35.854810 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:35.854814 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:35.854814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:35.854877 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:35.855101 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:35.855102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:35.855103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:35.855290 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:35.855425 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:35.855655 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 06:19:35.855739 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":134,"threads_started":1}
I20260926 06:19:35.856289 25228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:35.848489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33652 (local address 127.2.110.253:34687)
0926 06:19:35.848540 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:35.848542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:35.848559 (+ 17us) server_negotiation.cc:407] Connection header received
0926 06:19:35.848643 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:35.848643 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:35.848653 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:35.848718 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:35.849141 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:35.849978 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:35.850742 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:35.850958 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:35.851049 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:35.853438 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:35.853439 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:35.853439 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:19:35.853446 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:35.855046 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:35.855416 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:35.855417 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:35.855417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:35.855473 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:35.855684 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:35.855685 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:35.855686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:35.855825 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:35.855868 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:35.856070 (+ 202us) server_negotiation.cc:299] Negotiation successful
0926 06:19:35.856155 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 06:19:35.856576 25302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:35.848365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.253:34687 (local address 127.0.0.1:33652)
0926 06:19:35.848502 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:35.848517 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:35.848574 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:35.848729 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:19:35.848730 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:19:35.848959 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:19:35.849041 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:35.849043 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:35.850109 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:35.850111 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:35.850627 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:35.850629 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:35.851057 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:35.851059 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:35.851162 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:35.851631 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:19:35.851636 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:19:35.853321 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:19:35.855129 (+ 1808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:19:35.855129 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:19:35.855130 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:19:35.855343 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:19:35.855541 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:19:35.855542 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:19:35.855543 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:19:35.855602 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:19:35.855920 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:19:35.855921 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:19:35.855960 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 06:19:35.856375 (+ 415us) client_negotiation.cc:240] Negotiation successful
0926 06:19:35.856476 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260926 06:19:35.856962 25300 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.110.252" port: 42729 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 06:19:35 dist-test-slave-hndd krb5kdc[25068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403575, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.254@KRBTEST.COM for kudu/127.2.110.252@KRBTEST.COM
I20260926 06:19:35.860809 25232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:35.851570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.252:42729 (local address 127.0.0.1:55056)
0926 06:19:35.851615 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:35.851628 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:35.851682 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:35.851980 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:19:35.851982 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:19:35.852629 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:19:35.852706 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:35.852709 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:35.853617 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:35.853618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:35.854549 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:35.854551 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:35.854994 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:35.854995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:35.855068 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:35.855599 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:19:35.855604 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:19:35.857374 (+ 1770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:19:35.859782 (+ 2408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:19:35.859783 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:19:35.859784 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:19:35.859964 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:19:35.860180 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:19:35.860181 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:19:35.860181 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:19:35.860239 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:19:35.860480 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:19:35.860481 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:19:35.860528 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 06:19:35.860609 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 06:19:35.860712 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260926 06:19:35.860827 25305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:35.851656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55056 (local address 127.2.110.252:42729)
0926 06:19:35.851837 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:35.851839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:35.851847 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:19:35.851867 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:35.851868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:35.851879 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:35.851957 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:35.852785 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:35.853530 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:35.854634 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:35.854934 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:35.854995 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:35.857474 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:35.857475 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:35.857475 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:19:35.857482 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:35.859707 (+ 2225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:35.860046 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:35.860047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:35.860048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:35.860116 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:35.860307 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:35.860308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:35.860308 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:35.860414 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:35.860510 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:35.860633 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 06:19:35.860710 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":105,"threads_started":1}
Sep 26 06:19:35 dist-test-slave-hndd krb5kdc[25068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403575, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.252@KRBTEST.COM for kudu/127.2.110.252@KRBTEST.COM
Sep 26 06:19:35 dist-test-slave-hndd krb5kdc[25068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.110.254@KRBTEST.COM
I20260926 06:19:35.863121 25149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdd3f8d1dd0b4bf8aa62d31067735c4b: Bootstrap starting.
I20260926 06:19:35.864115 25149 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cdd3f8d1dd0b4bf8aa62d31067735c4b: Neither blocks nor log segments found. Creating new log.
I20260926 06:19:35.864537 25149 log.cc:824] T 00000000000000000000000000000000 P cdd3f8d1dd0b4bf8aa62d31067735c4b: Log is configured to *not* fsync() on all Append() calls
I20260926 06:19:35.865280 25302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:35.857180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.252:42729 (local address 127.0.0.1:55070)
0926 06:19:35.857220 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:35.857229 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:35.857262 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:35.857802 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:19:35.857803 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:19:35.857992 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:19:35.858028 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:35.858030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:35.858801 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:35.858802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:35.859380 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:35.859381 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:35.859874 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:35.859875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:35.859920 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:35.860362 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:19:35.860367 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:19:35.861857 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:19:35.864071 (+ 2214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:19:35.864072 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:19:35.864072 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:19:35.864464 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:19:35.864683 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:19:35.864684 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:19:35.864684 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:19:35.864735 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:19:35.864960 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:19:35.864960 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:19:35.864999 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 06:19:35.865062 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 06:19:35.865159 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260926 06:19:35.865409 25149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdd3f8d1dd0b4bf8aa62d31067735c4b: No bootstrap required, opened a new log
I20260926 06:19:35.865685 25304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:35.857435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55070 (local address 127.2.110.252:42729)
0926 06:19:35.857706 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:35.857708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:35.857715 (+ 7us) server_negotiation.cc:407] Connection header received
0926 06:19:35.857732 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:35.857733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:35.857741 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:35.857788 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:35.858116 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:35.858730 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:35.859482 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:35.859771 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:35.859841 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:35.862007 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:35.862009 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:35.862010 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:19:35.862021 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:35.863974 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:35.864540 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:35.864541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:35.864541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:35.864597 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:35.864781 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:35.864782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:35.864783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:35.864895 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:35.864967 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:35.865508 (+ 541us) server_negotiation.cc:299] Negotiation successful
0926 06:19:35.865585 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252}
I20260926 06:19:35.866707 25152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:35.856815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43082 (local address 127.2.110.254:43265)
0926 06:19:35.856854 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:35.856856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:35.856891 (+ 35us) server_negotiation.cc:407] Connection header received
0926 06:19:35.856966 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:35.856966 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:35.856976 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:35.857026 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:35.857682 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:35.858437 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:35.859453 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:35.859750 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:35.859817 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:35.863254 (+ 3437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:35.863256 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:35.863256 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:19:35.863265 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:35.864991 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:35.865689 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:35.865690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:35.865690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:35.865749 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:35.866013 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:35.866014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:35.866014 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:35.866156 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:35.866330 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:35.866539 (+ 209us) server_negotiation.cc:299] Negotiation successful
0926 06:19:35.866625 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 06:19:35.866876 25300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c6e5b98856445f2b999747ff1f36425: Bootstrap starting.
I20260926 06:19:35.867676 25300 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c6e5b98856445f2b999747ff1f36425: Neither blocks nor log segments found. Creating new log.
I20260926 06:19:35.867794 25149 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdd3f8d1dd0b4bf8aa62d31067735c4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd3f8d1dd0b4bf8aa62d31067735c4b" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 43265 } } peers { permanent_uuid: "778f32bf95a648d5bff3275adb49ceb9" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 34687 } } peers { permanent_uuid: "2c6e5b98856445f2b999747ff1f36425" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 42729 } }
I20260926 06:19:35.867928 25149 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdd3f8d1dd0b4bf8aa62d31067735c4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdd3f8d1dd0b4bf8aa62d31067735c4b, State: Initialized, Role: FOLLOWER
I20260926 06:19:35.867980 25300 log.cc:824] T 00000000000000000000000000000000 P 2c6e5b98856445f2b999747ff1f36425: Log is configured to *not* fsync() on all Append() calls
I20260926 06:19:35.868009 25149 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdd3f8d1dd0b4bf8aa62d31067735c4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd3f8d1dd0b4bf8aa62d31067735c4b" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 43265 } } peers { permanent_uuid: "778f32bf95a648d5bff3275adb49ceb9" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 34687 } } peers { permanent_uuid: "2c6e5b98856445f2b999747ff1f36425" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 42729 } }
I20260926 06:19:35.868251 25307 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdd3f8d1dd0b4bf8aa62d31067735c4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd3f8d1dd0b4bf8aa62d31067735c4b" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 43265 } } peers { permanent_uuid: "778f32bf95a648d5bff3275adb49ceb9" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 34687 } } peers { permanent_uuid: "2c6e5b98856445f2b999747ff1f36425" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 42729 } } }
I20260926 06:19:35.868330 25307 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdd3f8d1dd0b4bf8aa62d31067735c4b [sys.catalog]: This master's current role is: FOLLOWER
I20260926 06:19:35.868331 25149 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdd3f8d1dd0b4bf8aa62d31067735c4b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:19:35.868739 25300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c6e5b98856445f2b999747ff1f36425: No bootstrap required, opened a new log
W20260926 06:19:35.870741 25319 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cdd3f8d1dd0b4bf8aa62d31067735c4b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 06:19:35.870872 25319 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 06:19:35.871137 25300 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c6e5b98856445f2b999747ff1f36425 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd3f8d1dd0b4bf8aa62d31067735c4b" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 43265 } } peers { permanent_uuid: "778f32bf95a648d5bff3275adb49ceb9" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 34687 } } peers { permanent_uuid: "2c6e5b98856445f2b999747ff1f36425" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 42729 } }
I20260926 06:19:35.871284 25300 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c6e5b98856445f2b999747ff1f36425 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c6e5b98856445f2b999747ff1f36425, State: Initialized, Role: FOLLOWER
I20260926 06:19:35.871359 25300 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c6e5b98856445f2b999747ff1f36425 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd3f8d1dd0b4bf8aa62d31067735c4b" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 43265 } } peers { permanent_uuid: "778f32bf95a648d5bff3275adb49ceb9" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 34687 } } peers { permanent_uuid: "2c6e5b98856445f2b999747ff1f36425" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 42729 } }
I20260926 06:19:35.871649 25321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c6e5b98856445f2b999747ff1f36425 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd3f8d1dd0b4bf8aa62d31067735c4b" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 43265 } } peers { permanent_uuid: "778f32bf95a648d5bff3275adb49ceb9" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 34687 } } peers { permanent_uuid: "2c6e5b98856445f2b999747ff1f36425" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 42729 } } }
I20260926 06:19:35.871733 25321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c6e5b98856445f2b999747ff1f36425 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 06:19:35.871987 25300 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c6e5b98856445f2b999747ff1f36425 [sys.catalog]: configured and running, proceeding with master startup.
Sep 26 06:19:35 dist-test-slave-hndd krb5kdc[25068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.110.253@KRBTEST.COM
I20260926 06:19:35.871202 25081 master_runner.cc:186] Error getting master registration for 127.2.110.253:34687: OK, instance_id { permanent_uuid: "778f32bf95a648d5bff3275adb49ceb9" instance_seqno: 1790403575669680 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 06:19:35.874060 25081 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 06:19:35.878646 25228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:35.868402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33662 (local address 127.2.110.253:34687)
0926 06:19:35.868448 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:35.868450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:35.868460 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:19:35.868486 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:35.868487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:35.868499 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:35.868555 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:35.869288 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:35.870068 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:35.871000 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:35.871298 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:35.871334 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:35.874175 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:35.874176 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:35.874177 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:19:35.874186 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:35.876363 (+ 2177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:35.876815 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:35.876816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:35.876817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:35.876882 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:35.877334 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:35.877336 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:35.877336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:35.877585 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:35.877873 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:35.878430 (+ 557us) server_negotiation.cc:299] Negotiation successful
0926 06:19:35.878530 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 06:19:35.875631 25231 master_runner.cc:186] Error getting master registration for 127.2.110.253:34687: OK, instance_id { permanent_uuid: "778f32bf95a648d5bff3275adb49ceb9" instance_seqno: 1790403575669680 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 06:19:35.880021 25231 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 06:19:35.880223 25332 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2c6e5b98856445f2b999747ff1f36425: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 06:19:35.880265 25332 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 06:19:35.880396 2491 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:19:35.881199 25123 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "cdd3f8d1dd0b4bf8aa62d31067735c4b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43082
I20260926 06:19:35.881342 25123 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdd3f8d1dd0b4bf8aa62d31067735c4b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 06:19:35.881448 25123 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdd3f8d1dd0b4bf8aa62d31067735c4b [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:19:35.882251 25123 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdd3f8d1dd0b4bf8aa62d31067735c4b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd3f8d1dd0b4bf8aa62d31067735c4b" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 43265 } } peers { permanent_uuid: "778f32bf95a648d5bff3275adb49ceb9" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 34687 } } peers { permanent_uuid: "2c6e5b98856445f2b999747ff1f36425" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 42729 } }
I20260926 06:19:35.882484 25123 leader_election.cc:288] T 00000000000000000000000000000000 P cdd3f8d1dd0b4bf8aa62d31067735c4b [CANDIDATE]: Term 1 election: Requested vote from peers 778f32bf95a648d5bff3275adb49ceb9 (127.2.110.253:34687), 2c6e5b98856445f2b999747ff1f36425 (127.2.110.252:42729)
I20260926 06:19:35.882758 25274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cdd3f8d1dd0b4bf8aa62d31067735c4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2c6e5b98856445f2b999747ff1f36425"
I20260926 06:19:35.882879 25274 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c6e5b98856445f2b999747ff1f36425 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:19:35.883773 25274 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2c6e5b98856445f2b999747ff1f36425 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdd3f8d1dd0b4bf8aa62d31067735c4b in term 1.
I20260926 06:19:35.883977 25099 leader_election.cc:302] T 00000000000000000000000000000000 P cdd3f8d1dd0b4bf8aa62d31067735c4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c6e5b98856445f2b999747ff1f36425, cdd3f8d1dd0b4bf8aa62d31067735c4b; no voters:
I20260926 06:19:35.884117 25307 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdd3f8d1dd0b4bf8aa62d31067735c4b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:19:35.884285 25307 raft_consensus.cc:686] T 00000000000000000000000000000000 P cdd3f8d1dd0b4bf8aa62d31067735c4b [term 1 LEADER]: Becoming Leader. State: Replica: cdd3f8d1dd0b4bf8aa62d31067735c4b, State: Running, Role: LEADER
I20260926 06:19:35.884383 25307 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdd3f8d1dd0b4bf8aa62d31067735c4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd3f8d1dd0b4bf8aa62d31067735c4b" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 43265 } } peers { permanent_uuid: "778f32bf95a648d5bff3275adb49ceb9" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 34687 } } peers { permanent_uuid: "2c6e5b98856445f2b999747ff1f36425" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 42729 } }
I20260926 06:19:35.884838 25340 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdd3f8d1dd0b4bf8aa62d31067735c4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdd3f8d1dd0b4bf8aa62d31067735c4b. Latest consensus state: current_term: 1 leader_uuid: "cdd3f8d1dd0b4bf8aa62d31067735c4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd3f8d1dd0b4bf8aa62d31067735c4b" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 43265 } } peers { permanent_uuid: "778f32bf95a648d5bff3275adb49ceb9" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 34687 } } peers { permanent_uuid: "2c6e5b98856445f2b999747ff1f36425" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 42729 } } }
I20260926 06:19:35.885110 25340 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdd3f8d1dd0b4bf8aa62d31067735c4b [sys.catalog]: This master's current role is: LEADER
I20260926 06:19:35.893507 25152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:35.883876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43096 (local address 127.2.110.254:43265)
0926 06:19:35.884328 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:35.884330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:35.884473 (+ 143us) server_negotiation.cc:407] Connection header received
0926 06:19:35.884617 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:35.884618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:35.884630 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:35.884686 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:35.885563 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:35.886342 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:35.887491 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:35.887796 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:35.887865 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:35.890118 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:35.890120 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:35.890120 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:19:35.890128 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:35.891749 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:35.892314 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:35.892315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:35.892316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:35.892373 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:35.892689 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:35.892690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:35.892690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:35.892829 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:35.892917 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:35.893283 (+ 366us) server_negotiation.cc:299] Negotiation successful
0926 06:19:35.893386 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429}
I20260926 06:19:35.895383 25343 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:19:35.895653 25343 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:19:35.899338 25304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:35.884396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55080 (local address 127.2.110.252:42729)
0926 06:19:35.884438 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:35.884440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:35.885623 (+ 1183us) server_negotiation.cc:407] Connection header received
0926 06:19:35.885827 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:35.885828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:35.885838 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:35.885867 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:35.888330 (+ 2463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:35.889026 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:35.891878 (+ 2852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:35.892197 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:35.892240 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:35.895039 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:35.895040 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:35.895041 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:19:35.895048 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:35.896916 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:35.897388 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:35.897389 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:35.897389 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:35.897446 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:35.897651 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:35.897652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:35.897652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:35.897795 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:35.897898 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:35.899145 (+ 1247us) server_negotiation.cc:299] Negotiation successful
0926 06:19:35.899236 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 06:19:35.900270 25228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:35.884081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33674 (local address 127.2.110.253:34687)
0926 06:19:35.884193 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:35.884195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:35.885678 (+ 1483us) server_negotiation.cc:407] Connection header received
0926 06:19:35.887517 (+ 1839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:35.887518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:35.887531 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:35.887563 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:35.890126 (+ 2563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:35.890838 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:35.892562 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:35.892875 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:35.892914 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:35.895012 (+ 2098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:35.895013 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:35.895014 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:19:35.895021 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:35.897766 (+ 2745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:35.898228 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:35.898229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:35.898229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:35.898283 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:35.898463 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:35.898464 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:35.898464 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:35.898765 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:35.899008 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:35.900050 (+ 1042us) server_negotiation.cc:299] Negotiation successful
0926 06:19:35.900140 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 06:19:35.902412 25274 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2c6e5b98856445f2b999747ff1f36425 [term 1 FOLLOWER]: Refusing update from remote peer cdd3f8d1dd0b4bf8aa62d31067735c4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:19:35.902796 25307 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cdd3f8d1dd0b4bf8aa62d31067735c4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c6e5b98856445f2b999747ff1f36425" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 42729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:19:35.904042 25224 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.110.252" port: 42729 } attempt: 3
I20260926 06:19:35.904383 25321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c6e5b98856445f2b999747ff1f36425 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdd3f8d1dd0b4bf8aa62d31067735c4b. Latest consensus state: current_term: 1 leader_uuid: "cdd3f8d1dd0b4bf8aa62d31067735c4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd3f8d1dd0b4bf8aa62d31067735c4b" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 43265 } } peers { permanent_uuid: "778f32bf95a648d5bff3275adb49ceb9" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 34687 } } peers { permanent_uuid: "2c6e5b98856445f2b999747ff1f36425" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 42729 } } }
I20260926 06:19:35.904477 25321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c6e5b98856445f2b999747ff1f36425 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 06:19:35.905915 25321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c6e5b98856445f2b999747ff1f36425 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cdd3f8d1dd0b4bf8aa62d31067735c4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd3f8d1dd0b4bf8aa62d31067735c4b" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 43265 } } peers { permanent_uuid: "778f32bf95a648d5bff3275adb49ceb9" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 34687 } } peers { permanent_uuid: "2c6e5b98856445f2b999747ff1f36425" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 42729 } } }
I20260926 06:19:35.905915 25307 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdd3f8d1dd0b4bf8aa62d31067735c4b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cdd3f8d1dd0b4bf8aa62d31067735c4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd3f8d1dd0b4bf8aa62d31067735c4b" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 43265 } } peers { permanent_uuid: "778f32bf95a648d5bff3275adb49ceb9" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 34687 } } peers { permanent_uuid: "2c6e5b98856445f2b999747ff1f36425" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 42729 } } }
I20260926 06:19:35.905998 25321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c6e5b98856445f2b999747ff1f36425 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 06:19:35.905998 25307 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdd3f8d1dd0b4bf8aa62d31067735c4b [sys.catalog]: This master's current role is: LEADER
I20260926 06:19:35.906208 25344 mvcc.cc:205] Tried to move back new op lower bound from 7333493046893322240 to 7333493046823194624. Current Snapshot: MvccSnapshot[applied={T|T < 7333493046893322240}]
I20260926 06:19:35.906955 25345 mvcc.cc:205] Tried to move back new op lower bound from 7333493046893322240 to 7333493046823194624. Current Snapshot: MvccSnapshot[applied={T|T < 7333493046893322240}]
I20260926 06:19:35.907553 25343 catalog_manager.cc:1347] Generated new cluster ID: d7e71ec18ca44806bc3d91da2307cf9f
I20260926 06:19:35.907608 25343 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 06:19:35 dist-test-slave-hndd krb5kdc[25068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403575, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.110.253@KRBTEST.COM for kudu/127.2.110.252@KRBTEST.COM
I20260926 06:19:35.913606 25304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:35.904465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55082 (local address 127.2.110.252:42729)
0926 06:19:35.904541 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:35.904543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:35.904551 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:19:35.904571 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:35.904572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:35.904581 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:35.904652 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:35.904981 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:35.905776 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:35.907138 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:35.907440 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:35.907503 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:35.909823 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:19:35.909825 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:19:35.909826 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:19:35.909835 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:19:35.912360 (+ 2525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:35.912765 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:35.912766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:35.912766 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:35.912820 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:19:35.912998 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:19:35.912999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:19:35.912999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:19:35.913128 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:19:35.913185 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:35.913414 (+ 229us) server_negotiation.cc:299] Negotiation successful
0926 06:19:35.913494 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 06:19:35.913916 25226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:35.904373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.110.252:42729 (local address 127.0.0.1:55082)
0926 06:19:35.904421 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 06:19:35.904436 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 06:19:35.904492 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:19:35.904657 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:19:35.904658 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:19:35.904838 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:19:35.904891 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:35.904893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:35.905942 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:35.905943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:35.906765 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:19:35.906767 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:19:35.907571 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:19:35.907572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:19:35.907633 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:35.908143 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:19:35.908149 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:19:35.909711 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:19:35.912459 (+ 2748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:19:35.912460 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:19:35.912460 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:19:35.912686 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:19:35.912884 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:19:35.912885 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:19:35.912885 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:19:35.912937 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:19:35.913210 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:19:35.913211 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:19:35.913276 (+ 65us) client_negotiation.cc:769] Sending connection context
0926 06:19:35.913713 (+ 437us) client_negotiation.cc:240] Negotiation successful
0926 06:19:35.913821 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260926 06:19:35.915676 25224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 778f32bf95a648d5bff3275adb49ceb9: Bootstrap starting.
I20260926 06:19:35.916369 25224 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 778f32bf95a648d5bff3275adb49ceb9: Neither blocks nor log segments found. Creating new log.
I20260926 06:19:35.916658 25224 log.cc:824] T 00000000000000000000000000000000 P 778f32bf95a648d5bff3275adb49ceb9: Log is configured to *not* fsync() on all Append() calls
I20260926 06:19:35.917649 25224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 778f32bf95a648d5bff3275adb49ceb9: No bootstrap required, opened a new log
I20260926 06:19:35.919487 25224 raft_consensus.cc:348] T 00000000000000000000000000000000 P 778f32bf95a648d5bff3275adb49ceb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd3f8d1dd0b4bf8aa62d31067735c4b" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 43265 } } peers { permanent_uuid: "778f32bf95a648d5bff3275adb49ceb9" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 34687 } } peers { permanent_uuid: "2c6e5b98856445f2b999747ff1f36425" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 42729 } }
I20260926 06:19:35.919636 25224 raft_consensus.cc:729] T 00000000000000000000000000000000 P 778f32bf95a648d5bff3275adb49ceb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 778f32bf95a648d5bff3275adb49ceb9, State: Initialized, Role: FOLLOWER
I20260926 06:19:35.919706 25224 consensus_queue.cc:260] T 00000000000000000000000000000000 P 778f32bf95a648d5bff3275adb49ceb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd3f8d1dd0b4bf8aa62d31067735c4b" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 43265 } } peers { permanent_uuid: "778f32bf95a648d5bff3275adb49ceb9" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 34687 } } peers { permanent_uuid: "2c6e5b98856445f2b999747ff1f36425" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 42729 } }
I20260926 06:19:35.920022 25224 sys_catalog.cc:564] T 00000000000000000000000000000000 P 778f32bf95a648d5bff3275adb49ceb9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:19:35.920853 25191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cdd3f8d1dd0b4bf8aa62d31067735c4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "778f32bf95a648d5bff3275adb49ceb9"
I20260926 06:19:35.920964 25191 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 778f32bf95a648d5bff3275adb49ceb9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:19:35.921960 25191 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 778f32bf95a648d5bff3275adb49ceb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdd3f8d1dd0b4bf8aa62d31067735c4b in term 1.
I20260926 06:19:35.922356 25198 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 778f32bf95a648d5bff3275adb49ceb9 [term 1 FOLLOWER]: Refusing update from remote peer cdd3f8d1dd0b4bf8aa62d31067735c4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:19:35.922612 25340 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cdd3f8d1dd0b4bf8aa62d31067735c4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "778f32bf95a648d5bff3275adb49ceb9" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 34687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:19:35.924057 25343 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:19:35.924846 25343 catalog_manager.cc:1504] Loading token signing keys...
W20260926 06:19:35.927294 25365 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 778f32bf95a648d5bff3275adb49ceb9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 06:19:35.927423 25365 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 06:19:35.927592 25364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 778f32bf95a648d5bff3275adb49ceb9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cdd3f8d1dd0b4bf8aa62d31067735c4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd3f8d1dd0b4bf8aa62d31067735c4b" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 43265 } } peers { permanent_uuid: "778f32bf95a648d5bff3275adb49ceb9" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 34687 } } peers { permanent_uuid: "2c6e5b98856445f2b999747ff1f36425" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 42729 } } }
I20260926 06:19:35.927724 25364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 778f32bf95a648d5bff3275adb49ceb9 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 06:19:35.927798 25364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 778f32bf95a648d5bff3275adb49ceb9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdd3f8d1dd0b4bf8aa62d31067735c4b. Latest consensus state: current_term: 1 leader_uuid: "cdd3f8d1dd0b4bf8aa62d31067735c4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd3f8d1dd0b4bf8aa62d31067735c4b" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 43265 } } peers { permanent_uuid: "778f32bf95a648d5bff3275adb49ceb9" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 34687 } } peers { permanent_uuid: "2c6e5b98856445f2b999747ff1f36425" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 42729 } } }
I20260926 06:19:35.927876 25364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 778f32bf95a648d5bff3275adb49ceb9 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 06:19:35.927932 25364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 778f32bf95a648d5bff3275adb49ceb9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cdd3f8d1dd0b4bf8aa62d31067735c4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd3f8d1dd0b4bf8aa62d31067735c4b" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 43265 } } peers { permanent_uuid: "778f32bf95a648d5bff3275adb49ceb9" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 34687 } } peers { permanent_uuid: "2c6e5b98856445f2b999747ff1f36425" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 42729 } } }
I20260926 06:19:35.928006 25364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 778f32bf95a648d5bff3275adb49ceb9 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 06:19:35.930110 25362 mvcc.cc:205] Tried to move back new op lower bound from 7333493046970855424 to 7333493046823194624. Current Snapshot: MvccSnapshot[applied={T|T < 7333493046893322240}]
I20260926 06:19:35.933634 25307 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdd3f8d1dd0b4bf8aa62d31067735c4b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cdd3f8d1dd0b4bf8aa62d31067735c4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd3f8d1dd0b4bf8aa62d31067735c4b" member_type: VOTER last_known_addr { host: "127.2.110.254" port: 43265 } } peers { permanent_uuid: "778f32bf95a648d5bff3275adb49ceb9" member_type: VOTER last_known_addr { host: "127.2.110.253" port: 34687 } } peers { permanent_uuid: "2c6e5b98856445f2b999747ff1f36425" member_type: VOTER last_known_addr { host: "127.2.110.252" port: 42729 } } }
I20260926 06:19:35.933739 25307 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdd3f8d1dd0b4bf8aa62d31067735c4b [sys.catalog]: This master's current role is: LEADER
I20260926 06:19:35.935456 25343 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cdd3f8d1dd0b4bf8aa62d31067735c4b: Generated new TSK 0
I20260926 06:19:35.935654 25343 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 06:19:35.963528 25228 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:19:35.960882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33690 (local address 127.2.110.253:34687)
0926 06:19:35.960931 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:35.960933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:35.962035 (+ 1102us) server_negotiation.cc:407] Connection header received
0926 06:19:35.962608 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:35.962610 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:35.962626 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:35.962662 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:35.963430 (+ 768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33690: BlockingRecv error: recv got EOF from 127.0.0.1:33690 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260926 06:19:35.964784 25304 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:19:35.961645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55098 (local address 127.2.110.252:42729)
0926 06:19:35.961727 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:35.961729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:35.964091 (+ 2362us) server_negotiation.cc:407] Connection header received
0926 06:19:35.964203 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:35.964204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:35.964218 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:35.964252 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:35.964704 (+ 452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55098: BlockingRecv error: recv got EOF from 127.0.0.1:55098 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260926 06:19:35.966187 25152 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:19:35.962252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43108 (local address 127.2.110.254:43265)
0926 06:19:35.962305 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:35.962306 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:35.965253 (+ 2947us) server_negotiation.cc:407] Connection header received
0926 06:19:35.965558 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:35.965560 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:35.965577 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:35.965630 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:35.966102 (+ 472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43108: BlockingRecv error: recv got EOF from 127.0.0.1:43108 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260926 06:19:35.969414 25228 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:19:35.968470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33698 (local address 127.2.110.253:34687)
0926 06:19:35.968517 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:35.968519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:35.968764 (+ 245us) server_negotiation.cc:407] Connection header received
0926 06:19:35.968890 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:35.968891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:35.968915 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:35.968946 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:35.969320 (+ 374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33698: BlockingRecv error: recv got EOF from 127.0.0.1:33698 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 06:19:35.972841 25228 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:19:35.971890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33714 (local address 127.2.110.253:34687)
0926 06:19:35.971928 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:35.971931 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:35.972242 (+ 311us) server_negotiation.cc:407] Connection header received
0926 06:19:35.972376 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:35.972377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:35.972388 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:35.972419 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:35.972785 (+ 366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33714: BlockingRecv error: recv got EOF from 127.0.0.1:33714 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 06:19:35.976956 25228 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:19:35.976081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33720 (local address 127.2.110.253:34687)
0926 06:19:35.976128 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:35.976130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:35.976390 (+ 260us) server_negotiation.cc:407] Connection header received
0926 06:19:35.976490 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:35.976491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:35.976502 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:35.976531 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:35.976898 (+ 367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33720: BlockingRecv error: recv got EOF from 127.0.0.1:33720 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260926 06:19:35.984422 25228 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:19:35.982866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33728 (local address 127.2.110.253:34687)
0926 06:19:35.982912 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:35.982914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:35.983538 (+ 624us) server_negotiation.cc:407] Connection header received
0926 06:19:35.983690 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:35.983691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:35.983707 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:35.983741 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:35.984261 (+ 520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33728: BlockingRecv error: recv got EOF from 127.0.0.1:33728 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 06:19:35.995173 25228 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:19:35.994195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33730 (local address 127.2.110.253:34687)
0926 06:19:35.994249 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:35.994252 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:35.994537 (+ 285us) server_negotiation.cc:407] Connection header received
0926 06:19:35.994647 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:35.994648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:35.994661 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:35.994699 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:35.995095 (+ 396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33730: BlockingRecv error: recv got EOF from 127.0.0.1:33730 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 06:19:36.013723 25228 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:19:36.012687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33736 (local address 127.2.110.253:34687)
0926 06:19:36.012732 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:36.012734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:36.013024 (+ 290us) server_negotiation.cc:407] Connection header received
0926 06:19:36.013165 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:36.013166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:36.013179 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:36.013208 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:36.013662 (+ 454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33736: BlockingRecv error: recv got EOF from 127.0.0.1:33736 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 06:19:36.048331 25228 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:19:36.047322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33744 (local address 127.2.110.253:34687)
0926 06:19:36.047370 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:36.047372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:36.047671 (+ 299us) server_negotiation.cc:407] Connection header received
0926 06:19:36.047785 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:36.047787 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:36.047800 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:36.047832 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:36.048266 (+ 434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33744: BlockingRecv error: recv got EOF from 127.0.0.1:33744 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260926 06:19:36.114789 25228 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:19:36.114105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33758 (local address 127.2.110.253:34687)
0926 06:19:36.114257 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:36.114260 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:36.114268 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:19:36.114297 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:36.114298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:36.114309 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:36.114358 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:36.114721 (+ 363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33758: BlockingRecv error: recv got EOF from 127.0.0.1:33758 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
W20260926 06:19:36.244855 25228 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:19:36.244061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45782 (local address 127.2.110.253:34687)
0926 06:19:36.244135 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:36.244137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:36.244245 (+ 108us) server_negotiation.cc:407] Connection header received
0926 06:19:36.244342 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:36.244343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:36.244355 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:36.244429 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:36.244788 (+ 359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45782: BlockingRecv error: recv got EOF from 127.0.0.1:45782 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260926 06:19:36.502734 25228 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:19:36.502150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45790 (local address 127.2.110.253:34687)
0926 06:19:36.502224 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:36.502226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:36.502238 (+ 12us) server_negotiation.cc:407] Connection header received
0926 06:19:36.502291 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:36.502292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:36.502302 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:36.502343 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:19:36.502663 (+ 320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45790: BlockingRecv error: recv got EOF from 127.0.0.1:45790 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 06:19:36.880760 25332 catalog_manager.cc:1259] Loaded cluster ID: d7e71ec18ca44806bc3d91da2307cf9f
I20260926 06:19:36.880874 25332 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2c6e5b98856445f2b999747ff1f36425: loading cluster ID for follower catalog manager: success
I20260926 06:19:36.881736 25332 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2c6e5b98856445f2b999747ff1f36425: acquiring CA information for follower catalog manager: success
I20260926 06:19:36.882052 25332 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2c6e5b98856445f2b999747ff1f36425: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 06:19:36.927948 25365 catalog_manager.cc:1259] Loaded cluster ID: d7e71ec18ca44806bc3d91da2307cf9f
I20260926 06:19:36.928087 25365 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 778f32bf95a648d5bff3275adb49ceb9: loading cluster ID for follower catalog manager: success
I20260926 06:19:36.928967 25365 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 778f32bf95a648d5bff3275adb49ceb9: acquiring CA information for follower catalog manager: success
I20260926 06:19:36.929299 25365 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 778f32bf95a648d5bff3275adb49ceb9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 06:19:37.018401 25434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:37.015801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45802 (local address 127.2.110.253:34687)
0926 06:19:37.015926 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:37.015928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:37.015937 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:19:37.015981 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:37.015982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:37.016038 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:37.016110 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:19:37.016303 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:37.016799 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:37.017599 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:37.017818 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:37.017881 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:37.018092 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:19:37.018203 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:19:37.018271 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:37.018315 (+ 44us) server_negotiation.cc:299] Negotiation successful
0926 06:19:37.018334 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":65,"threads_started":1}
I20260926 06:19:37.022979 25440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:37.020335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33568 (local address 127.2.110.252:42729)
0926 06:19:37.020476 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:37.020479 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:37.020488 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:19:37.020573 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:37.020574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:37.020625 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:37.020711 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:19:37.020825 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:37.021307 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:37.022099 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:37.022379 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:37.022447 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:37.022611 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:19:37.022731 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:19:37.022811 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:37.022861 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 06:19:37.022884 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":59,"threads_started":1}
I20260926 06:19:37.027307 25434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:37.024583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45804 (local address 127.2.110.253:34687)
0926 06:19:37.024628 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:37.024630 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:37.024786 (+ 156us) server_negotiation.cc:407] Connection header received
0926 06:19:37.024930 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:37.024931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:37.024944 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:37.025018 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:19:37.025138 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:37.025659 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:37.026494 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:37.026802 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:37.026869 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:37.027040 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:19:37.027073 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:19:37.027143 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:37.027194 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 06:19:37.027216 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 06:19:37.027828 25440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:37.024872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33578 (local address 127.2.110.252:42729)
0926 06:19:37.024917 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:37.024919 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:37.024926 (+ 7us) server_negotiation.cc:407] Connection header received
0926 06:19:37.024948 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:37.024949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:37.024958 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:37.025008 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:19:37.025203 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:37.025938 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:37.026947 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:37.027229 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:37.027288 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:37.027573 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:19:37.027601 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:19:37.027632 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:37.027727 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 06:19:37.027748 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 06:19:37.028031 25447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:19:37.024594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34274 (local address 127.2.110.254:43265)
0926 06:19:37.024707 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 06:19:37.024709 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:19:37.024858 (+ 149us) server_negotiation.cc:407] Connection header received
0926 06:19:37.024906 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:19:37.024907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:19:37.024960 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:19:37.025194 (+ 234us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:19:37.025232 (+ 38us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:37.026005 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:37.026871 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:19:37.027193 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:19:37.027289 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:19:37.027478 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:19:37.027602 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:19:37.027670 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 06:19:37.027928 (+ 258us) server_negotiation.cc:299] Negotiation successful
0926 06:19:37.027955 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":63,"threads_started":1}
I20260926 06:19:37.030056 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 25081
I20260926 06:19:37.035734 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 25156
I20260926 06:19:37.040555 2491 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgcQD_/build/release/bin/kudu with pid 25231
2026-09-26T06:19:37Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2876 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5733 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (284675 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST